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FOREWORD 


Command  and  control  (C;)  and  tactical  decision  making  are  growing  in  impor¬ 
tance  foi  both  the  wartime  Army  and  for  peacetime  training.  Decisions  and  the  com¬ 
mands  that  put  them  into  effect  mold  a  tactical  force  into  shape  lor  a  combat  operation. 
Decisions  made  about  execution  and  the  control  that  is  accomplished  maintain  and  refine 
the  form  and  substance  of  that  force.  The  U.S,  Army  Research  Institute  for  the  Behav¬ 
ioral  and  Social  Sciences  (ARI)  Fort.  Leavenworth  Field  Unit  continues  to  examine  tac¬ 
tical  decision  making,  This  study  looks  at  tactical  decision  making  in  the  context  of 
planning  a  battlefield  operation. 

The  Estimate  of  the  Situation  is  the  military’s  process  to  support  decision  making. 
One  major  subtask  of  the  Estimate  involves  course  of  action  (COA)  analysis,  selection, 
and  justification.  This  research  examines  staff  officer  performance  when  the  task  is 
structured  through  manual  job  aids  and  computer  information  management  tools.  These 
conditions  are  compared  to  a  case  where  procedures  go  unspecified  and  the  participants 
arc  free  to  use  their  own  approaches  for  COA  selection. 

The  research  was  performed  under  the  ARI  research  task  entitled  "Enhancing 
Command  Staff  Performance  in  Combat  Operations."  The  work  was  performed  in  ac¬ 
cordance  with  the  Memorandum  of  Agreement  with  the  Combined  Arms  Combat  Devel¬ 
opment  Activity  entitled,  "Development  and  Implementation  of  the  Future  Battle  Labo¬ 
ratory,"  dated  30  June  1989.  Initial  results  were  presented  to  the  Combined  Arms 
Command’s  Command  and  Control  Integration  Council  and  at  the  28th  Army  Operations 
Research  Symposium  during  the  fail  of  1989.  As  findings  from  this  research  became 
available,  they  were  transitioned  as  C2  lessons  learned  for  Operation  Desert  Shield 
("Winning  in  the  Desert  II:  Tactics,  Techniques,  and  Procedures  for  Maneuver  Com¬ 
manders,"  Center  for  Army  Lessons  Learned  Special  Edition  Newsletter  No.  90-8, 

Sep  90)  and  revisions  to  FM  101-5  ("Command  and  Control  for  Commanders  and  Staffs," 
Coordinating  Draft.  .Tun  901- 
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THE  EFFECTS  OF  PROCEDURAL  STRUCTURE  AND  COMPUTER  SUPPORT 
UPON  SELECTING  A  TACTICAL  COURSE  OF  ACTION 


EXECUTIVE  SUMMARY 


Requirement: 

It  is  generally  thought  that  there  are  three  ways  to  improve  decision  making: 
selection  of  good  decision  makers,  training  good  decision  makers,  and  aiding  the  process 
of  decision  making.  An  essential  ingredient  for  all  thiee  is  an  understanding  of  human 
decision-making  requirements.  The  U.S.  Army  is  interested  in  decisions  about  tactical 
courses  of  action.  The  Estimate  of  the  Situation  is  the  doctrinal  process  for  making 
tactical  decisions. 

Informal  observation  of  the  procedures  used  for  making  tactical  decisions  in  field, 
laboratory,  and  classroom  settings  led  to  the  researchers’  conclusion  that  there  is  a 
discrepancy  between  how  the  Estimate  is  described  in  training  and  field  manuals  and  how 
it  is  actually  performed.  Key  Army  individuals  have  acknowledged  that  Estimate  materi¬ 
als  arc  designed  for  an  officer  with  little  or  no  experience  in  higher  echelon,  tactical 
decision  making  and  with  unlimited  time  to  follow  the  process.  In  wartime  conflicts  and 
in  field  training  exercises,  there  is  little  time  to  complete  an  entire  Estimate.  Unfortu¬ 
nately,  there  are  not  standard  Estimate  materials  for  these  more  important  cases. 

The  goal  of  this  research  was  to  better  understand  tactical  decision  making  by 
looking  more  closely  at  one  phase  of  the  Estimate-selecting  and  justifying  a  course  of 
action.  The  three  primary  objectives  were  to  find  out 

•  the  extent  to  which  Estimate  procedures  are  followed, 

•  if  better  decisions  result  when  procedures  are  used,  and 

•  if  decisions  are  improved  when  they  are  aided  by  computers. 


Procedure: 

An  experiment  was  used  to  examine  the  objectives.  Two  participants  similar  in 
rank  to  those  on  division  operations  and  plans  staffs  and  with  Command  and  General 
Staff  Officer  College  education  were  teamed  together  to  perform  the  task.  The  team  was 
to  select  and  justify  a  course  of  action  (COA)  in  a  division-level  offensive  operation. 

Each  of  14  teams  was  assigned  to  one  of  three  experimental  conditions.  One  condition 
used  unspecified  procedures.  The  unspecified  condition  presented  the  team  with  the  goal 
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of  selecting  and  justifying  a  COA.  Scenario  information  was  available  in  notebooks  and 
on  maps  in  doctrinally  prescribed  formats  for  information,  such  as  higher  command’s 
warning  order,  situation  overlays,  previous  estimates,  unit  status  reports,  and  reference 
information.  The  second  condition  separated  the  task  into  nine  steps,  requiring  a  team 
to  follow  step  instructions  and  to  complete  work  sheets  associated  with  each  step.  The 
same  scenario  information  was  available  to  the  teams  in  this  structured  condition.  The 
third  condition  was  similar  to  the  structured  case,  but  instead  of  retrieving  information 
from  the  scenario  notebooks,  a  tactical  computer  information  system  was  used.  Also, 
instead  of  completing  work  sheets  as  in  the  structured  condition,  the  computer-supported 
teams  had  various  computer  word  processing,  map,  and  spreadsheet  tools  available  for 
some  of  the  steps. 

The  experiment  used  several  data  collection  methods  to  address  a  broad  range  of 
topics,  including  task  performance  (identification  of  facts,  force  arrays,  critical  events,  war 
gaming  COAs,  comparing  COAs,  and  justifying  selection  of  a  COA),  task  duration  times, 
early  decisions,  workload,  situational  awareness,  performance  process  (work  management, 
team  dynamics,  knowledge,  style),  task  assessment,  cognitive  abilities,  computer  experi¬ 
ence,  relative  importance  of  data,  human-machine  interface  design,  and  task-aiding 
possibilities.  An  expert  panel  solution  was  developed  over  a  period  of  months  before 
data  collection  and  was  used  in  scoring  and  comparing  participant  team  results  for  task 
performance  measures. 


Findings: 

Structured  and  computer-supported  teams  ranked  significantly  higher  than  the 
unspecified  teams  on  the  quality  of  their  COA  justification.  Unspecified  teams  did  not 
perform  the  task  in  the  same  way  as  teams  required  to  follow  the  procedural  Estimate 
guidance.  Unspecified  teams  left  out  steps,  did  not  perform  steps  in  as  objective  a 
manner  as  the  structured  teams,  and  repeated  steps.  Unspecified  teams  tended  to  refer 
more  to  standard  Estimate  training  materials,  apparently  seeking  procedural  guidance. 

There  was  no  additional  advantage  for  the  computer-supported  teams  over  the 
structured  teams  as  measured  by  quality  ranking.  Computer-supported  teams  that  used  a 
spreadsheet  tool  to  aggregate  numerical  factors  in  a  decision  matrix  avoided  any  arith¬ 
metic  errors;  each  manual,  structured  team  made  one  or  more  arithmetic  errors.  There 
were  no  significant  differences  in  time  or  workload  between  the  computer-supported  and 
structured  teams,  even  though  the  computer-supported  teams  only  had  about  2  hours 
exposure  to  the  newly  developed  tools  before  using  them. 

The  participants  had  difficulty  in  projecting  war  game  results  and  in  visualizing  the 
battle.  After  the  exercise,  most  participants  felt  that  additional  computer  support  would 
be  beneficial  for  the  task.  Performance  results  on  task  steps  identified  potential  targets 
for  additional  computer  support:  arraying  forces  *•>  ensure  that  all  combat  power  is 
taken  into  account;  identifying  and  understanding  critical  events  of  a  dynamic  operation; 
and  war  gaming  and  projecting  battle  results. 
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The  use  of  structured  procedures  in  both  the  structured  and  computer -supported 
conditions  identified  shortfalls  in  the  Estimate  process.  To  reduce  the  chance  of  biased 
analysis,  the  standard  Estimate  training  materials  recommend  that  a  COA  not  be  selected 
until  all  COAs  are  independently  evaluated  and  compared.  Thirteen  out  of  the  fourteen 
teams  came  to  a  conclusion  before  the  time  the  Estimate  indicates  is  appropriate.  No 
penalty  was  issued  against  those  teams  that  made  an  early  selection  and  no  primacy  bias 
was  evident  as  five  of  the  teams  eventually  switched  to  another  COA. 

The  structured  and  computer-supported  teams  were  required  to  compare  COAs 
following  a  decision  analytic  or  multiattribute  utility  approach.  In  this  approach,  each 
COA  was  assigned  values  on  objective  and  subjective  factors  by  the  team.  These  factors 
were  then  weighted  for  importance  and  summed  to  give  an  overall  figure  of  merit.  There 
was  very  little  difference  between  the  two  COAs  on  any  of  the  objective  factors  for  any 
of  the  teams.  The  teams  seemed  to  load  the  subjective  factors  so  the  outcome  would 
correspond  to  their  COA  "choice."  The  expert  panel’s  solution  also  tended  to  lack  much 
distinction  using  the  linear,  scaled,  and  weighted  approach  of  multiattribute  utility  theory. 
It  would  seem  that  an  a  priori  linear  model  of  factors  does  not  fit  well  with  the  highly 
interactive  and  probably  uncertain  aspects  of  the  COAs.  A  more  appropriate  approach 
to  distinguish  between  COAs  was  when  the  teams  looked  for  information  that  suggested 
a  COA  would  not  be  feasible  or  would  have  a  greater  chance  of  failing  than  another 
COA. 


Utilization  of  Findings: 

The  findings  have  been  used  to  highlight  the  existence  of  problems  in  the  Esti¬ 
mate  process:  certain  steps  are  not  performed  well  (e.g.,  arraying  forces);  there  are 
insufficient  means  to  perform  some  steps  (e.g.,  war  gaming);  and  a  linear,  decision 
analytic  model  for  selection  is  not  universally  applicable.  These  insights  about  the 
process  have  been  used  to  make  revisions  to  doctrinal  materials  on  staff  procedures  and 
in  C2  lessons  learned  provided  to  Desert  Shield  commanders.  The  results  of  this  study 
have  also  been  used  to  identify  human  performance  enhancement  concepts  for  a  proto¬ 
type  decision  aid  called  the  Operations  Planning  Tools  (OPT).  OPT  is  being  considered 
for  implementation  into  tactical  data  systems. 
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THE  EFFECTS  OF  PROCEDURAL  STRUCTURE  AND  COMPUTER  SUPPORT 
UPON  SELECTING  A  TACTICAL  COURSE  OF  ACTION 


INTRODUCTION 
The  Problem 

AirLand  Battle  doctrine  envisions  a  dynamic  battlefield  requiring  initiative,  agility, 
synchronization,  and  operations  in  depth.  In  order  to  keep  pace  with  evolutionary 
changes  in  tactical  doctrine,  improvements  in  Army  command  and  control  (C2)  are 
required.  Problems  in  tactical  decision  making  stem  from  the  battlefield  environment, 
current  (^procedures,  and  human  capabilities  and  limitations  related  to  information 
processing  and  decision  making.  The  rapidly  changing  combat  environment  will  impose 
severe  time  pressures  on  the  staff  and  commander.  Enormous  amounts  of  information 
are  available  for  commanders  and  staff  officers,  and  there  is  little  time  to  process  this 
information  for  purposes  of  decision  making.  It  is  easy  for  humans  to  become 
overburdened  as  information  changes  rapidly  and  becomes  more  abundant.  Though 
there  is  an  abundance  of  data  and  information,  uncertainties  about  the  situation,  goals, 
and  outcomes  will  not  necessarily  be  diminished.  Time  pressures,  uncertainty,  stress,  and 
mental  fatigue  can  adversely  affect  human  performance. 

Procedures  for  making  tactical  decisions  are  described  in  Army  literature.  In  fact, 
the  Estimate  of  the  Situation  has  been  accepted  as  the  standard  for  decision  making 
since  1909  (ST  100-9,  1989).  Recent  versions  of  the  principal  training  documents  on  the 
Estimate  have  been  called  The  Command  Estimate  (ST  100-9,  1989).  The  commander’s 
estimate  is  defined  as 

"the  procedure  whereby  a  commander  decides  how  to  best  accomplish  the 
assigned  mission.  It  is  a  thorough  consideration  of  the  mission,  enemy, 
terrain  and  weather,  troops  available,  and  time  (METT-T)  and  other 
relevant  factors.  The  commander’s  estimate  is  based  on  personal 
knowledge  of  the  situation  and  on  staff  estimates."  (p  1-17,  FM  101-5-1,  Hq 
DA). 

More  recently  the  training  document  is  called  Techniques  and  Procedures  for  Tactical 
Decisionmaking  (ST  100-9,  1991).  This  report  will  refer  to  the  process  as  the  Estimate 
of  the  Situation  or,  just  simply,  the  Estimate. 

Despite  the  fact  that  the  procedures  for  the  Estimate  were  developed  to  be  logical 
and  analytical  (Michel,  1990),  staffs’  and  planners’  own  reports  indicate  that  not  all  steps 
are  followed.  Consequently,  only  a  portion  of  the  decision  making  procedure  may  be 
performed,  and  the  quality  of  the  decision  may  be  adversely  affected.  Overall,  staff 
officers  and  commanders  often  do  not  have  the  time  and  capabilities  to  analyze  courses 
of  action  in  a  systematic  manner.  The  application  of  a  systematic  approach  is  challenged 
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by  the  complexity  of  the  decision  situations  and  procedures,  relatively  infrequent 
opportunities  to  train  and  practice,  and  the  potential  for  lack  of  belief  in  the  usefulness 
of  the  procedures. 

One  important  consideration  is  to  determine  the  usefulness  of  the  procedures 
themselves.  The  Estimate  implies  that  a  decision  analytic  approach  (or  multi-attribute 
utility  theory)  is  used  for  selecting  among  courses  of  action.  The  problem  is  that  this 
approach  may  not  fit  all  tactical  problems.  Tactical  decision  making  is  very  complex  in 
contrast  to  typical  choice  problems  to  which  a  multi-attribute  utility  approach  have  been 
confidently  applied.  , 


Work  by  Klein  and  associates  (Klein,  Calderwood  &  MacGregor,  1989;  Klein  & 
MacGregor,  1988;  Thordsen,  Galushka,  Klein,  Young,  &  Brezovic,  1990)  and  earlier 
work  by  Simon  (1955)  suggest  that  the  best  procedures  are  naturalistic  ones.  Their  belief 
is  that  people  need  only  develop  a  workable  solution  that  satisfies  their  decision  criteria 
or  planning  goals.  Simon  refers  to  this  as  "satisficing,"  as  opposed  to  optimizing.  Klein 
takes  an  even  stronger  view  and  proposes  that  the  rapid  decision  making  that 
characterizes  experts  is  done  on  a  recognitional  basis.  To  an  experienced  decision 
maker,  cues  and  expectancies  set  an  immediate  association  to  an  appropriate  response. 


One  reason  that  the  decision  analytic  approach  may  not  always  fit  the  Estimate  is 
that  in  tactical  situations  the  real  problem  often  is  to  determine  what  the  goals  and 
criteria  are  and  to  arrange  them  into  concepts  to  accomplish  the  mission.  This  is  the 
broader  context  of  planning,  as  opposed  just  selecting  among  pre-established  options. 


The  Estimate  process  is  more  applicable  prior  to  a  conflict  when  there  are  relaxed 
time  demands.  But  once  a  tactical  operation  starts  there  are  no  established  procedures 
to  truncate  or  speed  up  the  steps.  Also  the  systematic  procedures  have  been  postulated 
to  be  ?  pplicable  only  to  officers  who  are  learning  the  process  as  a  student  in  a  formal 
course  or  as  a  newcomer  to  a  staff.  Further  it  has  been  speculated  that  the  Estimate  is 
too  rigid  and  proceduralized  for  experts  who  can  quickly  size  up  a  situation  and  who 
have  a  repertoire  of  responses  to  select  from  based  on  their  knowledge  of  similar  tactical 
situations.  Writers  of  doctrine  for  C2  have  suggested  that  the  Estimate  process  has  to  be 
changed  (or  something  different  created  to  augment  it)  to  apply  it  to  ongoing  battles 
where  rapid  planning  and  replanning  are  paramount  and  where  it  will  be  performed  by 
an  integrated  staff  with  different  specialties  and  experience  levels. 


Another  approach  to  address  performance  problems  with  the  Estimate  is  to  devise 
computer  assistance  or  aiding.  There  are  certain  activities,  like  retrieval  of  information 
from  large  data  stores  or  arithmetic  computations,  that  can  be  performed  more  quickly 
and  accurately  by  computers  while  humans  are  more  capable  in  other  areas.  A 
computer-based  system  could  be  used  to  relieve  some  of  the  staff  officer’s  workload. 

This  could  allow  the  staff  and  commander  to  analyze  a  wider  range  of  factors  with  the 
critical  factors  being  examined  in  greater  detail  and  depth. 
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Purpose  and  Rationale  of  the  Experiment 


To  conduct  research  to  assess  human  performance  in  tactical  decision  making,  the 
Army  Research  Institute  (ARI)  Field  Unit  at  Fort  Leavenworth  developed  a  laboratory 
test-bed  called  the  Experimental  Development,  Demonstration,  and  Integration  Center 
(EDDIC).  For  the  current  experiment,  the  EDDIC  equipment  and  facilities  were  used 
to  determine  the  effects  of  procedural  structure  and  computer  support  on  the 
performance  of  two  members  of  an  operations  plan  section  (G3)  who  are  required  to 
analyze  and  decide  upon  tactical  courses  of  action. 

i 

It  was  predicted  that  there  would  be  performance  differences  between  officers 
required  to  follow  the  estimate  process  and  officers  given  only  general  task  goals.  This 
latter  condition,  referred  to  as  manual  unspecif  ed  procedures  (or  unspecified),  was 
inserted  into  the  experiment  to  represent  conditions  under  which  staff  officers  in  the 
field  currently  plan.  After  receiving  training  in  the  estimate  of  the  situation  in  branch 
and  command  arid  staff  schools,  staff- officers  are  left  to  their  own  devices  to  apply  the 
procedures  in  the  field.  Certain  staffs  might  use  more  structured  procedures  than  other 
staffs.  By  not  specifying  what  procedures  to  follow  for  a  group  of  experimental  teams  it 
was  possible  to  observe  what  procedures  were  chosen  and  how  they  were  followed. 

We  were  unsure  whether  enforcing  a  structured  estimate  procedure  would  be 
better  or  worse  than  not  specifying  any  specific  set  of  procedures.  The  structured 
condition  might  be  better  if  the  officers  in  the  unspecified  condition  did  not  remember  a 
set  of  procedures  or,  if  they  made  irrational,  biased  decisions  which  the  systematic 
estimate  process  is  supposed  to  discourage.  The  structured  estimate  process  might  be 
worse  if  it  imposed  procedures  which  the  officers  did  not  accept  or  if  it  did  not  support 
the  dynamic  nature  of  their  tactical  decision  making. 

Structured  procedures  were  also  of  interest  because  in  order  to  apply  computer 
support  to  tactical  decision  making  and  planning,  there  must  be  some  level  of  structure 
and  consistency  in  procedures  and  reasoning  on  which  to  base  the  computer  assistance. 
When  this  experiment  was  first  contemplated,  it  was  an  accepted  belief  that  active  Army 
divisions  differed  in  the  way  they  did  planning  and  decision  making.  So  imposing 
structured  procedures  on  officers  given  the  task  of  analyzing  courses  of  action  serves 
both  as  a  necessary  precondition  and  a  comparison  to  a  computer-supported  condition. 

It  was  predicted  that  computer  support  supplemented  with  job  and  decision  aids 
could  reduce  the  level  of  difficulty  and  the  amount  of  time  associated  with  application  of 
a  structured  approach  to  tactical  derision  making.  This  is  predicted  in  turn  to  increase 
the  quality  of  the  decisions  and  the  decision  making  process.  The  derision  making 
procedures,  job  aids,  and  automated  support  were  integrated  into  the  Tactical  Planning 
Workstation  (Flanagan  and  Fallcsen,  1990)  and  Course  of  Action  Assessment  Tool 
(COAAT)  (Ross,  1989).  The  Workstation  and  COAAT  were  based  on  information, 
concepts,  and  procedures  presented  in  U.S.  Army  literature  (FM  101-5  and  ST  100-9). 
Experience  in  other  efforts  of  C2  automation  (Carter,  Archer  &  Murray,  1988;  Lussier, 
1986;  Michel  &  Riedel,  1988)  was  an  important  part  of  the  development  of  these  took. 
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Experimental  Treatment  Conditions 


The  three  experimental  groups  of  teams  are  called  unspecified,  structured,  and 
computer-supported.  Participants  in  the  unspecified  group  were  allowed  to  perform  the 
tactical  decision  making  task  using  techniques  and  procedures  of  their  choosing.  Field- 
related  experience  as  well  as  training  received  in  the  Command  and  General  Staff 
College  (CGSC)  were  undoubtedly  influential  factors.  Teams  in  the  structured  group 
were  provided  a  set  of  work  sheets  and  job  aids  that  encouraged  a  systematic  approach 
to  performing  certain  steps  of  the  task.  Step  sequence  and  job  aids  were  developed 
using  concepts  and  principles  outlined  in  Army  literature  (e.g.,  CGSC,  1989). 
Participants  in  the  computer-supported  group  used  the  same  procedures  as  did  the 
participants  in  the  structured  group;  however,  computer-support  was  available  to  assist 
with  certain  steps  of  the  task. 


Experimental  Hypotheses 

Different  outcomes  of  decision  quality  were  considered  possible  from  the  efforts 
of  teams  in  the  three  experimental  conditions.  It  is  difficult  to  predict  all  possible 
outcomes  and  process  findings  from  the  three  conditions.  Some  potential  outcomes  and 
interpretations  for  various  outcomes  follow. 

If  the  computer-supported  condition  produces  the  highest  quality  solutions,  then  it 
can  be  assumed  that  the  computer-supported  tools  provide  the  most  predominant 
benefit.  Computer-support  can  be  used  to  access  information,  to  perform  arithmetic 
calculations,  to  track  and  record  intermediate  steps,  and  so  foith.  Computers  can 
provide  rapid  storage  and  recall  of  information.  Computer  support  has  the  potential  to 
augment  performance  by  performing  manual  functions  automatically  and  more  accurately 
(e.g.,  arithmetic  calculations).  Computer  information  and  menu  selection  can  act  as 
reminders  to  the  user  to  be  complete  in  decision  making  considerations.  Since 
computers  work  with  chunks  of  information,  chunks  resulting  from  the  ongoing  decision 
making  process  can  be  stored  to  keep  a  record  of  preliminary  conclusions.  An  indirect 
benefit  of  computer-support  can  be  to  give  the  decision  makers  more  time  to  consider  a 
wider  range  of  information  and  to  allow  completion  of  all  steps  of  the  task. 

Computer-support  does  not  automatically  make  performance  faster;  computers  can 
take  overhead  in  terms  of  data  entry  and  control  which  manual  tasks  do  not.  If 
computer-supported  approaches  produce  poorer  solutions,  then  more  training  and 
familiarization  might  be  required.  Otherwise  poor  outcomes  may  be  due  to  the 
computer  tools  impeding  performance  because  of  their  inflexibility,  invalidity,  or  some 
other  characteristic.  If  the  computer-supported  solutions  are  no  worse  than  good 
solutions  from  either  or  both  of  the  other  conditions,  then  one  might  predict  that  after 
more  familiarization,  the  tools  will  improve  performance  to  a  significant  degree.  If  the 
computer-supported  tools  were  not  used  to  perform  the  task,  it  would  indicate  a  problem 
with  user  acceptance. 
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If  the  structured  condition  has  the  hignest  quality  solutions,  then  it  can  be 
concluded  that  structured  estimate  procedures  are  favorable  or  that  having  a  manual  job 
aid  to  guide  procedures  is  beneficial.  If  the  solutions  from  the  structured  condition  are 
worse  than  the  unspecified  condition,  then  this  would  suggest  that  the  estimate  was  not 
the  most  effective  set  of  procedures  for  the  problem.  Structured  procedures  might 
require  longer  times  to  perform  the  task  because  one  step  should  be  completed  before 
going  on  to  the  next  step.  The  steps  are  detailed  and  analytical  to  reduce  the  chance 
that  biases  will  creep  in  and  affect  the  conclusions.  Time  and  workload  measures  will  be 
compared  across  conditions  to  see  if  the  structured  procedures  require  higher  workload 
than  perhaps  the  more  flexible  unspecified  procedure^. 

If  the  unspecified  condition  has  the  highest  quality  solutions,  then  it  might  be 
concluded  that  staff  officers  can  efficiently  taiior  or  adapt  the  estimate  procedures  to  a 
problem  at  hand.  If  the  unspecified  condition  produces  better  solutions  than  the 
structured  conditions,  but  the  officers  in  the  unspecified  condition  u»e  procedures  which 
are  very  close  to  the  structured  ones,,  then  this  suggests  that  the  mere  presence  of 
detailed  guidance  would  detract  from  performance.  If  the  unspecified  condition 
produces  better  solutions  with  procedures  different  from  the  structured  ones,  then  those 
new  procedures  can  be  used  as  candidate  improvements. 


DacaamIi  T«ai«a« 
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This  experiment  addressed  many  additional  research  issues.  The  additional  issues 
were  used  for  diagnosis  to  understand  any  resulting  differences  among  unspecified, 
structured,  and  computer-supported  performance.  Dealing  with  these  supplemental 
issues  allows  a  better  understanding  of  information  and  decision  making  processes 
involved  in  analyzing  and  determining  courses  of  action.  Interface  design  issues  related 
to  the  EDDIC  system  are  also  addressed.  The  following  summarizes  issues  and 
questions  addressed  in  this  experiment. 

•  What  computer  experience  do  the  participants  have? 

•  What  is  the  performance  of  the  structured  and  computer-supported  groups  on 
each  of  the  task  steps  compared  to  the  expert  panel. 

•  What  information  does  the  participants  use  in  making  a  decision?  What  is  the 
relative  importance  of  that  information? 

•  How  adept  are  the  participants  at  arraying  forces  and  calculating  combat  power 
values  and  ratios? 


•  Do  the  participants  appropriately  identify  and  deal  with  critical  events? 

•  How  do  participants  war  game  (especially  those  in  the  unspecified  group)?  Can 
those  using  specified  procedures  follow  the  procedures?  What  objective  and 
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subjective  factors  are  considered  important  enough  to  use  in  war  gaming?  How 
sensitive  are  the  war  game  estimates?  What  confidence  do  participants  have  in 
their  war  game  estimates? 

•  How  error-prone  are  the  participants’  arithmetic  calculations? 

•  How  are  conclusions  and  selections  made  and  justified?  How  thorough  are 
they?  Are  they  logical  and  consistent? 

•  How  well  can  the  groups  adhere  to  the  benchmarks  for  task  and  step  times? 

•  Do  the  structured  and  computer-support;  <.  -oups  follow  the  structured  steps  or 
do  they  jump  ahead  and  back,  tend  to  be  bia>  and  arrive  at  conclusions  before 
the  proper  step? 

•  How  do  the  teams  organize  and  manage  their  work? 

•  What  kind  of  relationships  are  there  between  individuals  on  a  team?  Do  the 
relationships  affect  performance? 

•  What  level  of  knowledge  do  the  participants  have  about  the  task  and  tactics? 

•  What  style  of  performance  (motivation,  level  of  detail)  do  the  participants 
follow?  What  are  characteristics  of  the  decision  process  (degree  of  analysis, 
stability,  amount  of  confirmation)? 


•  How  much  have  the  participants  perceived,  learned,  and  retained  about  the 
scenario  (mission,  enemy,  terrain,  own  troops,  and  time  available)?  Can 
incomplete  understandings  be  tracked  to  performance  outcomes? 


Is  there  a  difference  among  types  of  workload?  Does  workload  differ  for 
Does  workload  differ  across  conditions? 


different  tasKsY 


•  What  comments  and  suggestions  do  the  participants  have  about  the  estimate 
task  (CO A  analysis)? 

•  What  suggestions  do  the  participants  have  for  supporting  the  task  with 
automation? 


•  What  is  the  level  of  problem  solving  abilities  (as  measured  by  the  Complex 
Cognitive  Assessment  Battery)  of  the  participants?  How  do  they  relate  to 
performance  on  the  task? 

•  How  usable  was  the  Tactical  Planning  Workstation  and  COAAT?  What 
features  were  most  desirable?  What  changes  should  be  made? 
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METHOD 


The  first  two  subsections  describe  the  selection  criteria  for  participants  in  the 
experiment  and  the  facilities  and  equipment  used  to  support  the  experiment.  The 
activities  to  develop  and  prepare  the  experiment  and  methods  were  extensive.  The  next 
section  describes  analytical  and  developmental  activities  that  were  required  10  identify 
the  experimental  task,  develop  the  tactical  scenario,  identify  the  experimental  conditions, 
develop  automated  tools,  and  determine  the  final  experimental  design  by  using  findings 
from  pilot  testing.  Experimental  procedures  are  described  in  a  methods  section,  and  the 
final  section  addresses  data  collection  and  analysis  procedures. 


Participants 

To  minimize  the  possibility  of  inter-subject  differences  in  knowledge  and 
experience  related  to  the  experimental  task,  the  following  qualifications  were  established 
for  participants: 

•  A  rank  of  major  or  higher, 

•  A  combat  arms  officer  (infantry,  armor,  and  field  artillery), 

•  A  graduate  of  the  branch  advanced  course  in  their  specialty  area, 

•  A  graduate  or  current  student  of  the  Command  and  General  Staff  Officers 

Course  (CGSOC). 

Participants  were  recruited  as  volunteers  from  officers  assigned  at  Fort 
Leavenworth.  Three  sources  of  officers  were  drawn  upon:  faculty,  staff,  and  students  at 
the  Command  and  General  Staff  College  (CGSC);  military  analysts  from  the  TRADOC 
Analysis  Command;  and  CGSOC  graduates  awaiting  order  from  the  Combined  Arms 
Command.  To  induce  participation,  officers  were  told  about  the  experiment  as  an 
opportunity  to  solve  military  tactical  problems  and  to  comment  on  decision  making 
procedures,  manual  job  aids,  and  computer  support. 

From  the  outset,  it  became  evident  that  the  total  number  of  participants  would  be 
unknown  because  of  sporadic  responses  to  the  recruiting  process.  A  total  of  14  two-man 
teams  were  recruited.  There  were  four  teams  for  the  unspecified  condition  and  five 
teams  each  for  the  structured  and  computer-supported  conditions.  The  background  and 
qualifications  of  participants  was  not  known  prior  to  their  arrival.  Because  of  the 
intricacies  of  the  computer  condition  and  the  extensive  paper  materials  required,  a 
treatment  condition  was  fixed  prior  to  the  arrival  of  the  participants.  The  participants 
could  not  be  assigned  to  the  conditions  based  on  matching.  A  preset  testing  order  (using 
balanced  random  strategy)  was  established  for  the  experimental  conditions.  As 
participants  became  available  they  were  assigned  to  each  successive  treatment  condition. 
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Equipment 


The  following  subsection  describes  the  laboratory  facilities  in  which  the 
experiment  was  conducted  ana  the  computer  systems  used  by  the  computer-supported 
group. 


Laboratory  Facilities 

l 

The  experiment  was  conducted  in  the  main  laboratory  of  the  Experimental 
Development,  Demonstration,  and  Integration  Center  (EDDIC),  a  research  facility  of  the 
ARI  Field  Unit  at  Fort  Leavenworth,  Kansas.  There  were  differences  in  the 
configuration  of  the  supporting  facilities  for  the  treatment  conditions  as  shown  in  Figures 
1  and  2;  however,  the  following  features  of  the  facility  supported  experimental  activities 
for  all  conditions. 


The  laboratory  provided  instrumentation  capabilities  to  record  the  activities  of 
participants.  As  a  point  of  reference,  the  map  board  on  the  North  wall  was  considered 
the  front  of  the  main  laboratory.  Two  video  cameras  were  permanently  mounted  on  the 
side  walls  (east  and  west)  of  the  main  laboratory.  These  two  cameras  were  remotely 
controlled  front  the  observation  room  to  pan  and  zoom  about  the  laboratory  while  video 
and  audio  recordings  were  made  on  video  cassette  recorders  located  in  the  observation 
room.  These  video  cameras  also  provided  input  to  monitors  in  the  observation  room. 
Audio  input  to  the  recorders  was  accomplished  from  both  fixed  microphones  in  the 
laboratory  and  wireless  microphones  attached  to  each  participant  These  microphones 
ensured  the  best  fidelity  of  the  audio  track  on  the  video  tape.  A  third  video  camera 
(with  microphone)  was  tripod-mounted  in  the  rear  or  south  end  of  the  main  laboratoiy. 
This  camera  backed  up  the  fixed  cameras  and  also  provided  recording  of  exercise  time 
plus  a  direct  view  of  exercise  activities  at  the  north  end  of  the  main  laboratory.  This 
third  camera  had  an  automatic  focus  but  no  panning  capability. 


Direct  observation  of  all  exercises  was  performed  by  experimenter  personnel.  A 
moderator  was  present  in  the  laboratory  with  the  participants  to  give  instructions  and 
guidance.  Observers  were  located  at  the  rear  of  the  laboratory  in  an  observation  room. 
One-way  windows  provided  unobtrusive  observation.  Direct  communication  between 
experimenter  personnel  in  the  observation  room  and  the  moderator  in  the  main 
laboratory  was  maintained  by  the  use  of  a  two-way  radio  communication  system. 
Communications  on  this  system  were  recorded  manually  in  a  moderator’s  log  in  the 
laboratory. 


The  equipment  facilities  also  contained  information  (e.g.,  maps)  and  work  spaces 
to  support  conduct  of  the  experimental  task  by  the  participants.  This  support  will  be 
described  later  in  this  section. 
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Figure  1 
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Experimental  Development,  Demonstration,  and  Integration  Center  (EDD1C)  Main 
Laboratory  (Manual  condition). 
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Figure  2,  Experimental  Development,  Demonstration,  and  Integration  Center  (EDDIC)  Main 
Laboratory  (Automated  Support  condition). 
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Computer  Systems  and  Network 


Participants  in  the  computer-supported  group  used  two  computet  work  nations  to 
support  performance  of  the  experimental  task  (see  Figure  2).  The  Tactical  Planning 
Workstation  software  was  hosted  on  a  Sun  3/160C  Color  Sunstation  (Sun  Microsystems, 
Inc.)  that  included  a  high  resolution  19-inch  color  monitor  (1152  X  900  pixels).  The 
software  was  programmed  in  Ada  and  C  languages.  A  UNIX  operating  system  was  used. 
Participants  used  a  keyboard  or  a  three-button  mouse  to  interface  with  the  Tactical 
Planning  Workstation.  The  decision  aid  called  the  Course  of  Action  Assessment  Tool 
(COAAT)  was  programmed  in  LISP  and  other  symbolic  languages  to  allow  fast  and 
efficient  operation.  COAAT  was  hosted  on  a  Symbolics  3640  workstation.  The 
Symbolics  workstation  had  a  high  resolution,  fully  bit-mapped  black  and  white  screen,  a 
keyboard,  and  a  three-button  mouse.  The  local  area  network  (LAN)  for  computer 
systems  was  an  Ethernet.  This  allowed  for  high-speed  data  exchange  between 
information  processing  equipment  in  a  moderately-sized  geographic  area. 


Development  of  the  Experimental  Setting  and  Support 
The  Experimental  Task 


Identification  of  the  experimental  task  was  based  On  an  extensive  analysis  that 
determined  critical  functions  performed  during  tactical  decision  making  at  the  division 
echelon  (Carter,  et  al,  1988).  The  content,  organization,  relative  importance,  and 
suitability  of  division-level  battle  staff  functions  was  analyzed  to  determine  how 


performance  enhancement  could  be  achieved  through  using  computer  support.  The  steps 
and  sequential  relationship  of  the  functions  were  developed  and  then  refined  based  on 
coordination  with  CGSC  which  is  responsible  for  the  development  of  Army  tactical 
doctrine  and  with  the  Combined  Arms  Combat  Development  Activity  (CACDA)  (now 
the  Combined  Arms  Command-Combat  Developments)  which  is  responsible  for  the  C? 
mission  area  for  the  Army. 


Based  upon  the  analysis,  the  task  of  analyzing  tactical  courses  of  action  (COA) 
was  determined  to  be  of  greatest  importance  in  tactical  staff  planning.  The  steps 
composing  the  task  are  diagrammed  in  Figure  3.  The  steps  conform  to  doctrine 
enunciated  in  Army  Field  Manual  101-5,  Staff  Organization  and  Operations.  (1984)  and 
to  material  contained  in  USA  CGSC  Student  Text  100-9,  The  Cgrpmand  Estimate. 
(1989)  for  the  process  referred  to  as  preparation  of  the  commander’s  estimate  of  the 
situation. 


Tactical  courses  of  action  are  often  identified  in  plans  or  orders  from  higher 
command  and/or  from  guidance  given  by  the  division  commander.  After  basic  tactical 
courses  of  actions  are  identified,  planners  must  (a)  develop  an  appreciation  of  the 
scenario  and  the  current  tactical  situation  and  (b)  identify  the  pertinent  facts  that  will  be 
considered  when  deciding  upon  the  preferred  course  of  action  (see  Step  2).  The  third 
step  is  to  identify  and  list  assumptions  used  to  fill  gaps  in  knowledge.  The  number  of 
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Step  1  Step  2  Step  3  Step  4  Step  5 


Figure  3.  Steps  of  the  experimental  task  of  analyzing  courses  of  action. 


assumptions  are  generally  kept  to  a  minimum  so  that  the  basic  task  of  operation 
planning  is  not  "assumed  away."  Assumptions  may  be  added,  modified,  or  deleted  as  the 
analysis  progresses,  and  each  assumption  should  be  validated  before  the  analysis  is 
completed. 

During  Step  4,  planners  array  and  task  organize  the  friendly  forces  to  perform  the 
tactical  mission  under  each  alternative  course  of  action.  The  force  for  each  course  of 
action  should  be  organized  with  sufficient  combat  power  to  perform  the  assigned  mission 
as  well  as  be  in  conformance  with  the  division  commander’s  guidance. 

In  Step  5,  planners  identify  and  list  critical  events  for  each  of  the  alternative 
tactical  courses  of  action.  Critical  events  are  defined  as  those  specified  or  implied  battle 
tasks  the  completion  of  which  are  essential  to  mission  accomplishment  and  which,  in  the 
judgment  of  the  planner,  required  detailed  analysis. 

Step  6  starts  the  war  gaming.  For  each  critical  event,  the  planners  visualize  the 
battle  and  estimate  an  outcome  for  critical  battlefield  measures  (e.g.,  friendly  casualties). 
After  war  gaming,  they  must  aggregate  or  sum  projected  outcomes  for  each  measure  for 
all  critical  events  (Step  7).  At  this  point,  the  team  compares  the  tactical  courses  of 
action  (Step  8)  and  identifies  the  major  advantages  and  disadvantages  of  each  course  of 
action  (Step  9).  As  a  general  rule,  advantages  and  disadvantages  that  are  common  to  all 
courses  of  action  should  be  disregarded  since  the  staff  is  in  the  process  of  distinguishing 
between  the  courses  of  action.  Similarly,  advantages  and  disadvantages  which  are 
insignificant  should  be  disregarded.  When  all  of  the  significant  advantages  and 
disadvantages  are  identified  and  considered,  a  decision  is  made. 
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Tactical  Scenario 


The  experimental  task  was  performed  in  the  context  of  a  tactical  scenario  which 
provided  participants  with  the  situation  and  constraints  for  decision  making.  The 
scenario  was  a  standard  ARI  scenario  depicting  a  division-level  offensive  operation  in 
Western  Europe  (Carter,  Ross,  &  Michel,  1989;  Fallesen,  Michel,  &  Carter,  1989).  The 
tactical  scenario  provided  realism  to  the  experimental  setting  and,  more  importantly,  was 
a  means  to  control  independent  variables  representing  a  complex  real-world 
environment.  At  the  time  of  development,  the  scenario  was  based  on  standard 
information  related  to  military  organization,  equipment,  and  tactical  doctrine  for 
simulated  friendly  and  enemy  forces.  The  scenario’s  geographic  and  climatic  data  were 
representative  for  the  area  of  operations  and  season  of  the  year.  A  mechanized  infantry 
division  was  organized  based  on  the  J-Series  table  of  organization  and  equipment 
(TO&E)  (October  86  TRADOC  update).  The  scenario  was  developed  to  accommodate 
exercise  participation  by  two  operations  (G3)  staff  planners. 

■m 

The  tactical  scenario  included  two  tactical  courses  of  action.  The  course  of  action 
with  the  main  attack  in  the  northern  portion  of  the  division  zone  is  identified  as  COA  N, 
and  the  course  of  action  with  the  main  attack  in  the  south  is  identified  as  COA  S.  In 
order  to  avoid  any  bias  which  might  arise  from  the  order  in  which  the  courses  of  action 
were  presented  to  the  participants,  for  half  of  the  teams  COA  1  was  identified  as  the 
course  of  action  with  the  main  attack  in  the  north  (counter-balanced  condition  A).  In 
the  other  half,  COA  1  was  identified  as  the  course  of  action  with  fhe  main  attack  in  the 
south  (counter-balanced  condition  B). 

The  COAs  were  generated  so  that  careful  analysis  by  a  team  was  required  to 
increase  the  likelihood  that  the  better  COA  would  be  selected.  COA  S  had  the  heavier 
concentration  of  enemy  troops  (see  Figure  4),  and  this  situation  could  easily  lead  to  a 
"gut  reaction"  that  COA  N  would  be  favorable.  Further  analysis  should  reveal  that  COA 
N  has  a  combination  of  terrain,  enemy  disposition,  and  lighting  conditions  which 
produces  an  unfavorable  situation.  At  one  point  in  COA  N,  friendly  units  would  have  to 
conduct  a  night  fiver  crossing  against  a  "dug  in"  enemy. 


Computer  SpppffrtJCflpabilitieg 

The  human  performance  requirements  of  the  experimental  task  were  critical 
determinants  of  the  types  of  computer  support  required  to  improve  the  timeliness  and 
quality  of  decision  making.  The  following  describes  tools  that  were  developed  and 
implemented. 

Tactual  Planning  Workstation  Tools.  The  Tactical  Planning  Workstation 
(Flanagan  &  Faliesen,  1990)  provided  tools  to  visualize  the  terrain  using  varied  types  of 
map  displays,  to  visualize  and  reposition  friendly  and  enemy  units  on  a  map  display,  to 
create  and  display  tactical  overlays,  to  access  doctrinal  reference  and  planning 
information,  to  access  situational  information,  to  create  staff  products,  and  to  visualize 
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the  organization  of  units.  The  following  summarizes  some  of  the  major  functional 
capabilities  provided  by  the  workstation. 

•  View  reference  data:  Tactical  situation  independent  information  from  field 
manuals,  technical  manuals,  tables  of  organization  and  equipment,  and  related 
sources. 


•  View  situation  data:  Tactical  situation  dependent  information  that  represents 
the  battlefield  situation. 

\ 

•  Build  products:  Formats  or  work  sheets  that  guide  the  development  of  text 
products  such  as  estimates  and  orders,  and  tools  to  support  building  of  overlays. 

•  View  terrain:  Terrain  representation  using  a  vegetation  background,  shaded 
relief,  and  elevation  banding. 

•  TOAST  (task  organization  and  status  tool):  Representation  of  task 
organization,  unit  status,  and  changes  to  task  organization. 


Course  of  Action  Assessment  Tool  (COAAT1.  The  COAAT,  which  resides  on  the 
Symbolics  workstation,  supports  war  gaining  of  alternative  courses  of  action  (Ross,  1989). 
COAAT  consists  of  three  modules  used  to  assist  pariieipaiiis  in  selecting  and  identifying 
the  critical  events  for  all  courses  of  action,  war  gaming  critical  events,  and  comparing 
alternative  courses  of  action.  These  capabilities  are  described  in  greater  detail  in  the 
section  that  addresses  participant  activities  during  conduct  of  the  experimental  task. 


ExpeiMeata)  .Conditions 

The  experiment  had  three  conditions.  The  conditions  were: 

•  Unspecified:  The  team  chooses  how  to  perform  the  experimental  task  with 
reference  and  situational  data  available  in  notebooks. 

•  Structured:  Experimental  task  is  performed  using  prescribed  steps  and  paper 
work  sheets  prescribed  by  the  experimental  protocol  with  reference  and  situational 
data  being  available  in  notebooks. 

•  Computer-supported :  Experimental  task  is  performed  using  prescribed  steps 
and  computer  work  sheets  and  spreadsheets  provided  by  COAAT.  Reference  and 
situation  data  are  accessible  using  the  Tactical  Planning  Workstation. 
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Pilot  Tests 


Once  the  preliminary  design  of  the  experiment  was  developed,  two  pilot  exercises 
were  conducted  using  computer-supported  conditions  to  verify  the  feasibility  of  the 
experimental  paradigm  and  agenda.  A  major  lesson  learned  was  that  participants  could 
not  be  given  unlimited  time  to  complete  the  various  steps  of  the  process.  As  a  result  of 
these  findings,  it  was  necessary  to  impose  time  limits  on  certain  steps  or  phases  of  the 
exercise  to  ensure  that  the  exercise  was  completed  in  one  working  day.  Imposing  time 
constraints  made  it  infeasible  to  test  the  experimental  hypothesis  dealing  with  time. 
Agendas  were  used  to  aid  experimenters  in  the  management  of  the  experiment. 

Separate  agendas  were  necessary  for  each  experimental  condition  since  conditions 
differed  in  level  of  structure  and  automation.  For  all  conditions,  each  team  was  involved 
in  experimental  activities  for  a  full  working  day  (seven  to  eight  hours). 


Experimental  Procedures 


Experimental  procedures  varied  with  each  condition;  however,  the  following 
general  sequence  was  followed.  In  the  morning,  a  participant  team  received  a  briefing 
on  the  purpose  of  the  experiment.  Participants  then  completed  a  demographic 
questionnaire  and  a  consent  form.  This  was  followed  by  a  two-hour  and  15-minute 
training  program  that  varied  across  experimental  conditions.  After  training,  participants 
began  the  experimental  task  of  analyzing  tactical  courses  of  action.  Participants  became 
familiar  with  the  exercise  data  base  (reference  and  situation)  after  which  a  workload 
assessment  was  conducted.  Following  a  lunch  break,  participants  completed  the 
experimental  task.  Post-exercise  data  were  collected.  Participants  completed  another 
workload  assessment,  a  situational  awareness  questionnaire,  a  COA  task  evaluation 
questionnaire,  and,  for  the  computer-supported  conditions,  a  human-machine  interface 
questionnaire.  The  exercise  concluded  with  a  debriefing  of  the  participants.  The 
following  is  a  more  detailed  description  of  procedures. 


Background  Questionnaire 

A  background  questionnaire  (see  Appendix  A)  was  completed  by  each  participant 
immediately  upon  reporting  for  the  experiment.  The  participant  was  asked  to  provide 
information  on  his  military  education  and  experience,  experience  with  computers  and 
other  computer-supported  systems,  and  amount  of  staff  training  and  experience. 


Training 

Performance  of  the  experimental  task  by  participants  required  various 
combinations  of  training  in  the  following  areas: 

•  Availability  and  organization  of  situational  and  reference  material. 
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•  Sequence  and  steps  of  the  experimental  task. 

•  Operation  and  use  of  job  aids  and  computer-support. 

The  computer-supported  condition  required  more  training  than  the  other  two 
conditions  to  prepare  participants  to  use  the  computer  to  support  the  task.  As  a  result,  it 
was  necessary  to  provide  comparable  cognitive  activities  for  the  teams  in  the  structured 
and  unspecified  conditions.  This  was  achieved  by  having  participants  conduct  computer 
based  exercises  using  the  Complex  Cognitive  Assessment  Battery  (CCAB)  (Analytical 
Assessments  Center,  1987).  , 

The  duration  of  the  pre-exercise  activities  for  all  conditions  was  constant 
(approximately  two  hours  and  15  minutes);  however,  the  content  varied  by  condition. 
Training  under  each  exercise  condition  is  described  in  the  subparagraphs  which  follow. 

■M 

Unspecified  Condition.  Participants  used  the  self-administered  CCAB  to  take  nine 
CCAB  subtests  for  a  two-hour  period.  Exercise  participants  were  then  trained  for  15 
minutes  on  the  availability  and  organization  of  situational  and  reference  material 
presented  in  large  loose-leaf-bound  notebooks.  The  information  was  organized  by  staff 
functional  area  (personnel,  intelligence,  operations,  and  logistics),  by  broad  data 
categories,  and  by  detailed  data  elements.  The  volume  was  indexed  for  rapid  retrieval  of 
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Structured  Condition.  In  this  condition,  the  participants  took  five  CCAB  subtests 
for  one  hour  and  fifteen  minutes.  Experimental  task-related  training  then  occurred  for 
the  next  hour.  Using  the  same  training  provided  in  the  unspecified  condition, 
participants  were  familiarized  with  the  availability  and  organization  of  situational  and 
reference  material. 

The  structured  participants  were  then  trained  on  the  steps  and  structure  of  the 
task.  The  task  structure  was  based  on  U.S.  Army  doctrinal  publications  and  as  taught  in 
the  Command  and  General  Staff  Officer  Course,  Training  (and  later,  task  performance) 
was  based  upon  a  series  of  explanations  of  successive  steps  in  the  task  and  upon  work 
sheets  used  with  each  analysis  step,  A  workbook  provided  participants  with  work  sheets 
and  a  description  of  how  to  conduct  each  step  of  th  i  structured  process.  A  sample 
workbook  is  included  in  Appendix  B.  As  the  training  ensued,  the  trainer  used  the 
workbook  as  a  point  of  reference  to  discuss  a  process  step,  as  well  as  situational  and 
reference  data  necessary  to  perform  the  step.  In  this  manner,  the  exercise  participant 
was  trained  in  an  integrated  fashion  in  both  the  process  and  the  materials  to  support 
process  performance. 

Computer-supported  Condition.  In  the  computer-supported  condition,  the 
participants  were  trained  in  the  same  manner  as  were  the  structured  teams.  ITiey  were 
also  trained  on  use  of  the  Tactical  Planning  Workstation  and  COAAT  embodied  in  the 
Symbolics  workstation.  They  were  trained  on  the  availability  and  organization  of 
situational  and  reference  material  in  the  Tactical  Planning  Workstation.  Training  on 
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situational  and  reference  data  bases  focused  on  accessing  information  using  walking 
menus  and  windows  embodied  in  the  Tactical  Planning  Workstation.  Many  of  the  work 
sheets  to  support  war  gaming  (Steps  6,  7,  and  8  of  the  experimental  task)  are 
represented  by  computer  spreadsheets  in  COAAT.  Training  for  the  computer-supported 
condition  required  approximately  two  hours  and  fifteen  minutes  with  much  time  being 
devoted  to  making  the  participants  familiar  and  comfortable  with  the  computer  system. 
These  participants  did  not  receive  exposure  to  CCAB. 


Conducting  the  Exercise  . 

Operation  briering.  Conduct  of  the  experimental  procedures  required  the 
remaining  portion  of  a  working  day  for  each  team.  At  the  beginning  of  the  day,  the 
team  was  briefed  on  strategic  and  tactical  aspects  of  the  scenario  and  of  the 
requirements  of  the  team.  The  briefing  was  also  given  in  written  form  to  ensure  that  all 
teams  received  the  identical  information.  The  participant  teams  were  given  an 
opportunity  to  ask  questions  about  the  introduction;  however,  answers  were  provided  in 
strict  conformance  with  the  scripted  introduction.  When  both  members  of  the 
participant  team  had  completed  reading  the  briefing,  a  questionnaire  was  administered 
to  the  team  members  to  ensure  that  they  had  retained  critical  information  of  the 
operation  briefing.  The  team  members  were  then  debriefed  by  the  experimenter,  who 
provided  the  correct  answers  to  the  questionnaire. 

The  experimenter  next  read  an  extract  of  the  division  commander’s  planning 
guidance,  and  he  answered  any  questions  relating  to  the  guidance.  The  division 
commander’s  guidance  contained  the  two  tactical  courses  of  action  which  the  participant 
team  was  to  use  for  the  experimental  test.  The  full  text  of  the  commander’s  guidance 
appeared  in  the  situational  data  base  for  the  participants  to  refer  to  later. 

Participants  under  the  unspecified  condition  were  not  required  to  use  a  specified 
sequence  of  procedures  to  analyze  tactical  COAs;  they  were  free  to  follow  any  structure 
they  desired  to  perform  the  experimental  task.  If  desired,  participants  of  the  unspecified 
condition  could  use  doctrinal  publications  provided  to  them.  The  participants  in  he 
unspecified  condition  were  only  required  to  perform  Step  9  of  the  structured  procedures 
described  below. 

Select  courses  of  action  for  analysis  (step  1>.  The  courses  of  action  were  included 
as  key  elements  of  the  division  commander’s  planning  guidance.  All  participants  were 
briefed  on  these  courses  of  action  by  the  experimenter  at  the  start  of  the  exercise.  The 
courses  of  action  could  also  be  viewed  by  participants  using  the  situation  data  base.  In 
the  experiment,  the  courses  of  action  were  provided  to  participants  as  a  means  to  ensure 
experimental  control.  Without  such  control,  a  variety  of  courses  of  action  could  be 
advanced  by  participants,  and  analyses  of  these  diverse  courses  of  action  would  be 
exceptionally  difficult,  if  not  impossible. 
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Division  Commander’s  Planning  Guidance 

As  for  courses  of  action,  I  want  a  strong  initial  attack  supported  amply  by  artillery,  CAS,  and  attack 
helicopters.  I  also  want  a  strong  reserve  positioned  and  prepared  to  quickly  exploit  our  successes. 

Consider  COA  1  as  two  brigades  On  line  with  the  main  attack  in  the  direction  GREBENAU  (NB3322),  down 
the  JOSSA  River  valley  to  crossings  of  the  FULDA  River  in  the  vicinity  of  N1EDERJOSSA  (NB4025),  and 
on  to  the  division  objective.  In  this  COA,  I  visualize  a  supporting  attack  on  the  axis  LAUTERBACH 
(NB2809)  -  SCHLTTZ  (NB3913)  -  BURGHAUN  (NB5116)  to  fa  the  enemy  in  position  and  to  seize  limited 
objectives. 

1 

For  COA  2,  consider  two  brigades  on  line  with  the  main  attack  in  the  direction  of  LAUTERBACH  - 
SCHLTTZ  -  BURGHAUN  and  on  to  the  division  objective.  In  COA  2, 1  visualize  a  supporting  attack  on 
the  axis  GREBENAU  -  NIEDERJOSSA  to  fa  the  enemy  in  position  and  to  seize  limited  objectives. 


Review  area  of  interest  and  gather  facts  (step  21.  Access  to  situation  and 
reference  data  bases  allowed  participants  to  "read  up"  on  the  tactical  problem. 

Depending  on  the  experimental  condition,  data  could  be  viewed  in  either  paper  or  on 
video  displays.  The  situation  data  base  provided  scenario-dependent  data  for  24-hour 
time  periods  and  at  selected  intermediate  times  as  a  means  to  represent  battle  progress. 
The  situation  data  base  included  force  and  resource  status  information  for  both  friendly 
and  enemy  forces  as  weii  as  graphical  and  geographical  information  correlated  in  time 
with  narrative  information.  While  performing  Step  2,  the  participant  team  developed  a 
list  of  pertinent  facts  that  were  recorded  on  a  work  sheet  (list  of  pertinent  facts  work 
sheet)  for  future  reference.  The  situation  and  reference  data  bases  could  be  revisited 
during  the  exercise,  and  pertinent  facts  could  be  added  to  the  list  by  the  participants  at 
any  time  throughout  the  experiment. 

List  assumptions  (step  3).  It  was  necessary  to  guide  and  to  maintain  some  control 
over  conduct  of  the  experimental  task.  For  this  reason,  a  list  of  assumptions  was 
provided  to  the  teams  and  the  rationale  for  the  selection  of  each  assumption  was 
supplied.  Participants  were  told  that  the  provided  assumptions  were  the  only 
assumptions  to  be  used  during  the  task.  The  participants  were  told  to  read  and 
understand  the  assumptions  and  were  to  incorporate  these  assumptions  into  execution  of 
the  experimental  task. 

Array  forces  for  each  COA  (step  41.  For  the  friendly  force,  units  were  arrayed  and 
task  organized  for  the  main  attack,  supporting  attack,  and  reserve.  In  performing  this 
step,  die  team  should  attempt  to  achieve  a  combat  power  ratio  between  friendly  and 
enemy  forces  for  the  initial  battle  that  will  provide  a  fifty  percent  probability  of  mission 
success1  as  well  as  conform  to  the  division  commander’s  guidance.  A  task  organization 
work  sheet  (structured  condition)  or  computer  spreadsheet  (computer-supported 


1For  example,  to  achieve  success  in  an  attack  against  an  enemy  in  a  hasty  defensive  posture,  the  combat 
power  ratio  between  attacker  and  defender  must  be  2.5:1  or  better. 
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Assumption 


Planning  Arsumptions 

Rationale 


1.  No  nuclear,  biological,  or  chemical  weapons 
to  be  used. 

2.  Bridges  over  the  Fulda  and  Haune  rivers 
will  not  be  intact  for  friendly  use. 

3.  14  TA  will  not  reinforce  in  less  than  72 
hours. 


4.  313  Sep  Mech  Bde  will  not  be  available 
initially  for  employment. 

•m 

5.  32  AD  attack  will  be  rapid  and  successful 


6.  6  US  Corps  will  occupy  enemy  on  the 
North. 

7.  10  CAA  will  reposition  the  33  GTD  tc 
counter  the  operations  of  the  32  AD. 


8.  Rain  expected  on  7  Sep  will  make  Fulda 
river  unfordable  for  1-2  days. 


1.  Neither  combatant  has  used  NBC  weapons  and  no 
indications  that  the  enemy  will  use  them. 

2.  The  enemy  in  a  hasty  defense/delay  will  destroy 
major  bridges  to  impose  maximum  delay. 

3.  Intelligence  indicates  that  the  14  TA  is  in  assembly 
areas  and  is  not  moving.  Preparation  and  movement 
will  require  at  least  72  hours. 

4.  Corps  Commander  has  currently  assigned  a  RACO 
mission  to  313,  but  they  may  be  available  later. 

5.  32  AD  is  making  the  10  (US)  Corps  main  attack  with 
significant  combat  power.  Their  success  will  prevent  an 
attack  on  16  MID  southern  flank. 

6.  6  (US)  Corps  will  fully  engage  the  enemy  on  16 
MID's  northern  flank. 

n  \i ;:«-u 
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the  south,  the  33  GTD  will  be  employed  there  and  will 
not  be  committed  against  16  MID. 

8.  Rain  on  1-2  Sep  raised  the  Fulda  River  level.  More 
rain  will  probably  make  it  unfordable. 


*  condition)  was  used  to  record  task  organization  or  force  allocation  and  the  associated 
levels  of  combat  power.  In  this  experiment,  task  organization  was  constrained  to  units 
with  combat  power  (i.e.,  maneuver  battalions,  attack  helicopter  forces,  and  supporting 
artillery  units).  The  team  next  calculated  the  current  combat  power  of  friendly  units  and 
allocated  the  friendly  combat  power  to  the  main  and  supporting  attacks  in  sufficient 
quantity. 

Determine  critical  events  (step  5).  From  an  analysis  of  the  current  battle 
situation,  the  participant  team  identified  critical  events  for  the  main  and  supporting 
attacks  for  the  two  COAs.  A  critical  event  assignment  work  sheet  or  a  computer 
spreadsheet  was  used  to  record  a  critical  event  identification  number,  the  type  of  critical 
event,  the  objective,  and  comments.  Participants  were  provided  with  a  list  of  candidate 
critical  event  types  (e.g.,  cross  river,  seize  objective,  penetrate  first  defensive  belt)  to  aid 
in  event  identification;  this  list  provided  a  job  performance  aid  for  identifying  critical 
events  and  allowed  for  greater  standardization  in  the  types  of  responses  from 
participants.  Participants  also  classified  critical  events  into  the  factors  of  mission,  enemy, 
terrain  and  weather,  troops,  and  time  available  (METT-T).  Each  critical  event  was 
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"circled"  either  on  the  map  board  with  a  grease  pencil  or  an  automated  tool  available 
within  the  Tactical  Planning  Workstation. 


War  gaming  critical  events  (step  61.  At  the  onset  of  this  step,  the  moderator 
provided  the  structured  and  computer-supported  teams  with  a  list  of  pertinent  facts,  an 
array  and  an  allocation  of  combat  and  combat-support  forces  for  each  course  of  action, 
and  a  list  of  critical  events  for  each  course  of  action.  These  materials,  together  with  the 
list  of  assumptions  provided  during  Step  3,  constituted  a  baseline  for  continuing  the 
experimental  task.  Without  these  experiment  controls,  the  variety  of  war  gaming 
alternatives  offered  by  the  teams  could  make  data  scqring  and  analysis  very  difficult,  if 
not  impossible.  The  following  description  of  Step  6  applies  only  to  the  structured  and 
computer-supported  conditions. 


Teams  were  asked  to  visualize  the  battle  for  each  critical  event  (for  main  and 
supporting  attacks  for  each  CO  A)  in  order  to  predict  battle  outcome  for  selected 
measures.  Eight  battle  outcome  measures  were  offered  to  the  participant  team,  and  any 
of  these  could  be  used  by  the  team  for  war  gaming.  The  measures  provided  to  the  teams 
were  friendly  and  enemy  personnel  losses,  friendly  and  enemy  equipment  loses  (major 
end  items),  percentage  of  POL  expended,  percentage  of  ammunition  expended,  FEBA 
movement  (km)  for  the  main  attack,  and  battle  duration  (hours)  for  the  main  attack. 
Teams  were  allowed  to  provide  battle  outcome  estimates  for  three  phases  of  each 


measure  for  a  critical  event:  actions  before  the  actual  critical  event,  actions  during  the 
critical  event,  and  actions  taken  upon  completion  of  the  critical  event  (e.g., 
consolidation). 


In  performing  the  war  gaming  activity,  the  participant  team  for  structured  and 
computer-supported  conditions  completed  either  critical  event  war  gaming  work  sheets  or 
computer  spreadsheets  (Module  2  of  COAAT),  for  each  critical  event  for  the  two  CO  As. 
The  work  sheet  identified  the  critical  event  being  gamed,  divided  the  critical  event  into 
three  standard  phases,  and  offered  the  eight  war  gaming  measures  for  assessment  of 
battle  outcome.  If  the  team  provided  battle  outcome  estimates  for  the  three  phases  of 
each  critical  event,  then  COAAT  automatically  summed  the  estimate  for  the  phases. 

The  team  had  to  perform  their  own  addition  for  the  structured  condition. 


Aggregate  and  scale  battle  results  for  each  COA  (step  7).  After  war  gaming,  the 
results  for  each  course  of  action  were  aggregated  and  scaled  using  the  war  gaming 
summary  work  sheet  or  a  computer  spreadsheet.  For  each  of  the  battle  outcome 
measures,  the  following  procedure  was  used.  The  team  first  summed  predictions  for  all 
critical  events  for  both  the  main  attack  and  supporting  attacks  for  each  COA.  The 
values  for  main  and  supporting  attacks  were  then  added  for  each  COA.  A  Scaling 
Factors  Table  was  then  used  to  rate  the  "goodness"  of  each  measure  using  a  scale  that 
ranged  from  1  to  9  where  high  values  were  considered  "acceptable  or  good"  results  and 
low  numbers  were  considered  "unfavorable"  results.  This  summation  and  scaling  process 
was  conducted  on  all  battle  outcome  measures  for  each  course  of  action.  Summation 
and  scaling  were  done  manually  for  the  structured  condition  while  COAAT  automatically 
performed  these  functions  under  the  computer-supported  condition. 
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Compare  courses  of  action  (step  81.  To  this  point,  the  team  has  predicted  battle- 
outcome  measures  that  could  be  objectively  measured  during  an  actual  combat  episode. 
The  team  is  now  at  the  point  of  considering  more  "subjective"  factors  that  could 
contribute  to  success  or  failure  of  a  mission.  Five  subjective  measures  were  provided  to 
the  team  by  a  COA  comparison  work  sheet  (manual  structured)  or  a  computer 
spreadsheet  (computer  support). "  These  job  performance  aids  also  provided  the 
capability  for  the  team  to  include  three  more  subjective  measures  if  so  desired.  The 
work  sheets  and  spreadsheets  also  contained  the  eight  objective  measures  considered 
during  Steps  6  and  7. 

v 

The  next  activity  in  this  step  was  to  independently  weight  both  battle-outcome 
(objective)  and  subjective  measures.  The  weight  assigned  should  reflect  the  relative 
influence  of  that  measure  for  each  course  of  action.  Objective  and  subjective  measures 
may  be  weighted  as  independent  categories,  or  they  may  be  lumped  as  a  single  category. 
The  measure  with  the  greatest  influence  on  the  selection  of  a  preferred  course  of  action 
as  determined  by  the  participants  was  weighted  highest.  All  other  factors  were  then 
weighted  in  relationship  to  the  most  important  factor  and  receive  a  weight  less  than  that 
factor. 


The  next  activity  in  this  step  was  to  scale  the  subjective  measures  for  goodness 
using  a  scaling  table  ihai  was  provided.  The  participant  scaled  only  those  subjective 
measures  which  were  considered  when  comparing  the  alternative  courses  of  action. 
Scaled  values  were  entered  in  the  COA  comparison  work  sheet.  COAAT  automatically 
performed  that  function  for  the  computer-supported  condition. 

Finally,  the  weight  and  scale  value  for  each  objective  and  subjective  measure  were 
multiplied.  Then  the  product  of  all  battle-outcome  measures  were  added  to  produce  a 
grand  sum  for  the  objective  measures.  A  similar  grand  sum  was  obtained  for  subjective 
measures.  Mathematical  computations  were  performed  automatically  by  Module  3  of 
COAAT  for  the  computer-supported  condition. 

Justify  recommended  COA  (step  9).  All  teams  used  a  COA  selection  and 
justification  work  sheet  to  record  advantages  and  disadvantages  associated  with  the  two 
COAs.  This  work  sheet  could  be  used  by  the  team  as  a  job  aid  in  making  their  decision. 
The  participant  teams  were  advised  to  analyze  and  evaluate  the  advantages  and 
disadvantages  by  a  technique  of  their  own  choosing  and  to  produce  a  narrative  which 
compares  the  courses  of  action.  The  narrative  should  have  been  in  sufficient  detail  to 
convince  the  commander  of  the  preferred  (recommended)  course  of  action. 


IMdoatLAssessment 

The  participants’  perception  of  workload  imposed  by  the  experimental  task  was 
assessed  using  the  NASA  Task  Load  Index  (NASA-TLX)  (NASA  Ames  Research 
Center,  1986).  This  assesses  mental  demand,  physical  demand,  temporal  demand, 
performance,  effort,  and  frustration  associated  with  task  performance.  Definitions  of 
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these  dimensions  of  workload  and  work  sheets  were  provided  to  participants.  In  each  of 
the  structured  exercises  (manual  and  computer-supported),  a  workload  assessment  form 
was  distributed  to  participants  twice  during  the  exercise,  once  after  performance  of  the 
first  three  planning  steps  and  again  at  exercise  completion.  In  the  unspecified  condition, 
workload  assessment  was  also  performed  twice;  however,  its  performance  was  not  linked 
to  specific  analytical  steps  of  the  task  but  was  scheduled  at  a  time  of  the  exercise 
comparable  to  when  workload  was  assessed  for  the  other  two  conditions. 


Post  Exercise  Procedures 


A  variety  of  performance  data  was  collected  following  conduct  of  the  experimental 
task.  The  following  subparagraphs  describe  data  collected  using  the  Situation  Awareness 
Questionnaire,  the  COA  Task  Evaluation  questionnaire,  the  Human-Machine  Interface 
questionnaire,  and  the  after-exercise  debriefing. 


Situation  awareness  questionnaire.  There  are  many  items  of  scenario  information 
in  the  situation  data  base  which  should  be  considered  by  the  participant  team  in 
performing  the  exercise.  The  objective  in  administering  this  questionnaire  was  to 
determine  the  extent  to  which  the  participants  have  an  adequate  and  accurate 
understanding  of  the  tactical  situation.  A  questionnaire  of  32  items  was  designed  to 
sample  the  extent  of  knowledge  of  key  tactical  information.  The  multiple  choice 
questionnaire  required  approximately  ten  minutes  to  complete.  A  copy  of  the  situation 
awareness  questionnaire  is  presented  and  summarized  in  Appendix  C. 


CQA.  anal.YSiS  task  evaluation-  The  COA  Analysis  Task  Evaluation  questionnaire 
was  used  to  examine  factors  related  to  how  the  participants  conducted  the  analysis  of 
courses  of  action.  All  participants  were  asked  for  their  priorities  of  information  sources, 
the  step  when  in  the  process  when  a  decision  was  made,  and  the  level  of  difficulty 
associated  with  the  various  steps  and  decision  making  activities.  Participants  in  the 
structured  and  computer-supported  conditions  were  asked  about  their  level  of  confidence 
in  estimates  of  war  gaming  measures  and  their  understanding  of  the  reason  for  scaling 
the  war  gaming  factors.  Appendix  D  presents  the  COA  Analysis  Task  Evaluation 
questionnaire. 


Human-machine  interface  evaluation.  This  questionnaire  (see  Appendix  E)  was 
presented  to  participants  exposed  to  the  computer-supported  condition.  Questions  were 
asked  on  issues  such  as  the  ease  of  access  to  data  sources,  the  most  useful  attribute  and 
features  of  the  map  display,  and  other  questions  related  to  case  of  operation  of  both  the 
Tactical  Planning  Workstation  and  COAAT. 


Team  profile.  A  team  profile  work  sheet  was  used  by  data  collectors  to  organize 
their  notes  related  to  work  management,  team  dynamics,  performance  style  and  strategy, 
performance  results,  and  use  of  media.  Notes  were  based  on  observation  of  participant 
performance  throughout  the  conduct  of  the  experimental  task.  Appendix  F  presents  the 
team  profile  work  sheet. 
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Performance  Measurement  and  Analysis 
The  Measurement  Challenge 

One  of  the  biggest  challenges  of  this  experiment  was  the  identification  or 
determination  of  performance  measures  and  criteria.  Analyzing  tactical  courses  of  action 
relies  heavily  on  information  management  and  cognitive  processing;  both  of  which  are 
difficult  to  quantify  and  assess.  There  is  a  prescribed  process  for  analyzing  courses  of 
action;  however,  the  outcomes  are  not  always  well-defined  and  standards  of  performance 
do  not  exist.  By  definition  courses  of  action  need  to  be  feasible;  if  not,  then  they  are 
dropped  from  consideration  and  no  longer  constitute  a  COA.  The  COAs  for  this 
experiment  were  developed  so  that  it  would  be  difficult  for  the  planners  to  distinguish 
between  them.  The  final  product  for  the  experimental  task  required  a  recommendation 
on  a  coursp  of  action;  however,  there  are  no  absolute  right  or  wrong  answers  to  this 
tactical  problem. 


Assessment  of  decision  making  processes  and  products  was  done  using  direct  and 
indirect  performance  measures.  Direct  measures  were  based  on  measurable 
performance  during  conduct  of  the  experimental  task;  these  measures  were  used 
primarily  to  test  research  hypotheses  related  to  the  quality  of  processes  and  products. 
Indirect  measures  were  based  on  data  collected  using  questionnaires  and  observation 
sheets  completed  by  participants  and  observers,  respectively.  Indirect  measures  were 
used  to  test  workload-related  hypotheses  and  secondary  research  issues,  and  to  provide 
insight  into  the  reasons  why  direct  measures  of  performance  came  out  as  they  did. 


Data  requirements  and  performance  measures  were  developed  for  process  steps 
and  the  final  products  for  the  tasks.  Use  of  an  expert  solution  to  the  tactical  problem 
was  the  primary  means  to  establish  a  baseline  level  of  performance  for  the  participants. 
The  following  provides  a  description  of  the  expert  solution;  direct  measures,  and  indirect 
measures  used  in  this  experiment. 


The -Expert-Sol  utton 

An  expert  solution  to  the  tactical  scenario  was  used  as  a  benchmark  from  which  to 
measure  the  performance  of  teams  participating  in  the  experiment.  Two  expert 
teams/panels  from  CGSC,  (each  team  consisting  of  two  subject  matter  experts) 
participated  in  the  development  of  an  initial  expert  solution.  The  expert  teams  actually 
had  to  solve  the  same  tactical  problem  presented  to  the  participants  in  the  experiment. 
Following  development  of  the  solution  by  the  expert  panels,  an  additional  panel 
consisting  of  military-experienced  contractor  personnel  and  a  psychologist  from  the  AR1 
Field  Unit  was  convened  to  fine-tune  the  expert  solution  to  meet  scoring  requirements  of 
the  experiment. 
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Even  though  the  expert  solution  was  used  as  a  benchmark,  certain  inherent 
limitations  arc  recognized.  The  expert  solution  is  just  one  possible  solution  developed  by 
a  small  number  of  "experts."  One  should  recognize  that  other  solutions  may  have  equally 
served  as  a  benchmark. 


Direct  Measures  of  Performance 

Direct  measures  of  task  processc  s  and  products  were  assessed  using  the  expert 
solution  as  a  benchmark  to  determine  (a)  the  number  *  (or  percentage)  of  item  matches 
with  the  expert  solution  and  (b)  the  amount  of  deviation  (i.e.,  a  difference  score)  of  the 
participant  team’s  score  from  the  expert  score.  The  following  describes  data  summary 
and  analysis  procedures  used  to  assess  decision  making  performance. 

Review  area  of  interest  and  gather  facts  (step  21.  For  analysis  of  Step  2,  the  list  of 
pertinent  facts  provided  by  the  participant  team  was  compared  and  evaluated  against  a 
similar  list  prepared  by  the  Expert  Panel.  The  expert  solution  identified  24  facts 
pertinent  to  the  exercise  performance.  The  pertinent  facts  selected  and  listed  by  the 
participant  team  were  grouped  by  METT-T  category  and  were  compared  to  the  expert 
solution  by  relating  key  words  and  phrases.  Each  fact  was  scored  as  a  match  or  no¬ 
match.  Each  METT-T  category  was  evaluated  for  matches;  however,  the  aggregate  score 
for  this  step  was  a  percentage  of  total  matches  by  the  participant  team.  The  scoring  for 
this  step  was  performed  by  the  expert  scorer.  In  the  scoring,  no  credit  was  given  for 
facts  in  addition  to  those  identified  by  the  expert  panel,  the  rationale  being  that  any 
other  facts  were  of  secondary  importance. 

Array  forces,  for  each  COA  (step  4).  Analysis  of  combat  power  (CP)  measures 
provided  an  indicator  of  how  effectively  the  team  arrayed  their  forces  to  achieve  an 
acceptable  probability  of  mission  success.  The  combat  power  allocation  resulting  from 
'  arraying  of  forces  by  the  expert  panel  was  used  as  the  point  of  reference  for  assessing 
how  effectively  the  participant  teams  arrayed  their  forces.  The  following  combat  power 
measures  for  the  friendly  force  were  calculated: 

(a)  Main  attack  difference  score  -  (expert  panel  main  attack  CP)  -  (participant  team  main  attack  CP) 

(b)  Supporting  attack  difference  score  *  (exper!  panel  supporting  attack  CP)  - 

(participant  team  supporting  attack  CP) 

(c)  Reserve  difference  score  »  (expert  panel  reserve  CP)  -  (participant  team  reserve  CP) 

(d)  Array  deficit  *  (total  available  CP)  -  (arrayed  CP) 

(e)  Total  array  error  «  |main  attack  difference  score]  +  [supporting  attack  difference  score]  + 

| reserve  difference  score  j 
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The  teams  arrayed  forces  for  the  main  attack,  the  supporting  attack,  and  the 
reserve  for  both  COAs;  however,  the  primary  focus  was  on  the  main  attack  (main  attack 
difference  score),  for  if  the  main  attack  did  not  succeed,  then  mission  would  probably 
not  succeed.  The  array  deficit  was  used  to  indicate  the  degree  to  which  the  team  used 
all  available  resources.  And  finally,  total  array  error  indicated  the  total  amount  of  error 
the  team  achieved  relative  to  the  expert  panel.  If,  in  arraying  the  forces,  the  participant 
team  simply  selected  and  arrayed  the  forces  and  did  not  compute  combat  power  scores 
and  combat  power  ratios,  the  scorer  determined  combat  power  allocations  resulting  from 
the  force  array  generated  by  the  team. 

Determine  critical  events ,i$tep  5).  Scoring  analyzed  the  critical  events  identified 
by  the  team  that  matched  critical  events  identified  in  the  expert  solution.  A  match 
required  the  participant  team’s  critical  event  to  both  (a)  be  of  the  same  description 
(type)  and  (b)  be  located  within  three  kilometers  of  center  of  mass  of  the  critical  event 
identified  |n  the  expert  solution.  Matches  and  non-matches  were  determined  for  each 
team  and  group  means  were  used  for- analysis. 

War  came  the  courses  of  action  (steps  6  and  71.  Eight  battle  outcome  measures 
were  presented  to  the  participant  team  to  assist  in  assessing  battle  outcome  of  each 
critical  event,  but  teams  were  not  required  to  consider  all  measures  during  war  gaming. 
For  purposes  of  scoring,  only  the  three  measures  corresponding  to  the  expert  solution 
were  scored  (the  other  five  measures  being  of  secondary  importance).  Two  of  the 
measures  scored  (friendly  equipment  losses  and  enemy  equipment  losses)  contributed 
directly  to  combat  power  because  they  represented  major  weapon  systems.  The  third 
measure  scored  was  battle  duration.  For  each  experimental  condition,  the  mean  and 
standard  deviation  of  predicted  measures  was  determined. 

Compare  courses  of  action  (step  8h  The  bases  for  comparing  courses  of  action 
are  the  weights  and  scales  assigned  to  each  measure.  Each  objective  and  subjective 
factor  analyzed  by  the  teams  was  assigned  a  weight  and  scaling  value.  For  each  objective 
factor,  multiplying  the  weight  by  the  scale  gave  a  single  score  for  that  factor.  The  sum  of 
products  for  all  objective  factors  gave  a  single  aggregate  measure  for  objective  factors.  A 
single  aggregate  measure  was  also  generated  for  subjective  factors. 

Justify  rec  mimended  COA  (step  91.  Several  scoring  approaches  were  considered 
for  analysis  of  the  decision  justification.  One  obvious  method  is  to  count  those  teams 
selecting  the  same  COA  as  the  experts  as  correct  and  the  teams  selecting  the  other  COA 
as  incorrect.  Although  the  selection  of  the  COA  was  important,  the  selection  itself  does 
not  sufficiently  distinguish  the  quality  of  reasoning.  Credit  for  only  the  correct  solution 
would  not  distinguish  between  those  teams  "chancing"  upon  the  expert-selected  COA  and 
those  appropriately  considering  many  aspects  of  the  situation  -  but  selecting  the  non¬ 
expert  COA  (because  of  attention  to  different  factors  of  importance). 

A  second  method  for  scoring  is  to  score  solutions  based  on  agreement  with  the 
factors  and  rationale  used  by  the  experts.  The  expert  solution  identified  specific 
advantages  and  disadvantages  of  each  course  of  action.  The  advantages  and 
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disadvantages  developed  by  the  participant  team  were  compared  by  the  expert  scorer  to 
those  of  the  expert  solution.  The  significant  factors  identified  in  the  expert  solution  were 
weighted  from  1  to  5  as  shown  in  Table  1  where  a  value  of  5  indicated  the  most 
significant  factors.  This  approach  to  scoring  is  similar  to  a  multi-attribute  utility 
approach. 

A  third  approach  focuses  on  the  merit  of  the  team’s  justifications  based  on  a  select 
set  of  factors.  The  reasons  for  choice  and  severity  of  errors  were  of  interest  for  the 
merit  of  justifications.  This  approach  focuses  on  the  essential  elements  distinguishing 
between  the  two  COAs  in  terms  of  feasibility  and  risk.  It  does  not  presume  that  the 
essence  of  the  comparison,  selection,  and  justification  can  be  adequately  done  as  an 
enumeration  strategy.  This  approach  considers  that  choices  involving  complex  options 
are  more  naturally  and  efficiently  made  by  elimination  by  aspects,  satisfying  minimal 
choice  criteria,  or  strategies  other  than  multi-attribute  utility. 

'4* 

All  three  approaches  were  used  and  are  reported  in  the  results  section.  The  major 
scoring  and  analysis  effort  went  toward  the  third  approach.  The 
rationale  used  by  the  experts  for  making  the  choice  boiled  down  to  the  following. 

The  greatest  distinction  in  the  feasibility  of  the  two  COAs  accomplishing  the 
mission  was  that  COA  N  reauired  an  ODDosed  river  crossing.  The  second  echelon. 

A  AM  ~  KS  - ~  —  ^ 

enemy  reserve  15  MRD,  would  be  expected  to  be  in  defensive  positions  on  the  far  side 
of  the  Fulda  River.  The  Fulda  crosses  the  Northern  avenue  and  was  unfordable  with 
only  limited  bridging  assets  available  to  the  friendly  forces.  In  the  South  the  Fulda  was 
fordable  and  the  enemy  second  echelon  would  have  to  move  out  of  their  defensive 
positions  to  encounter  the  friendly  forces.  While  moving,  the  enemy  could  be  engaged 
and  attrited  by  close  air  support  (CAS).  In  their  dug-in  positions  at  the  Fulda  they 
would  not  be  as  easily  attrited. 

These  significant  factors  were  operationalized  for  scoring  in  the  following  way.  r 
Teams  correctly  identifying  the  existence  of  the  second  echelon,  their  location,  and  their 
posture  (or  capability)  would  be  given  four  points.  If  a  team  recognized  that  they  were 
faced  in  the  North  with  a  river  crossing  over  an  unfordable  river  with  limited  bridging 
assets  they  were  given  3  additional  points.  The  second  echelon  opposition  force  was 
considered  to  be  more  critical  than  the  requirement  to  do  the  river  crossing.  Points  were 
reduced  foi  partial  assessments.  For  example,  if  teams  correctly  identified  the  existence 
of  the  force  but  misinterpreted  their  location  at  encounter  or  defensive  posture  the  score 
would  be  reduced  by  2  points.  The  recognition  of  limited  bridging  (at  a  minimum, 
implying  the  perceived  requirement  to  do  a  river  crossing  of  an  unfordable  river)  would 
attain  3  points.  If  the  river  was  identified  as  unfordable  then  2  points  were  awarded,  and 
if  only  the  need  to  do  the  crossing  was  identified  they  were  awarded  1  point.  Points  for 
opposition  and  river  crossing  considerations  were  summed  for  a  maximum  of  7  points. 
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Table  1 


The  expert  panel’s  weightings  for  critical  mission  factors  considered  during  COA 
justification 


METT-T  Factor 

Mission  Factor 

COA 

Favored 

Weight¬ 

ing 

Own  troops 

| 

River  crossing  operation 

n 

5 

Terrain 

Major  river  obstacle 

5 

Enemy 

Encounter  enemy  second  echelon 

South 

4 

Time 

Battle  duration 

South 

4 

Enemy 

Encounter  enemy  First  echelon 

North 

3 

Own  troops 

Major  friendly  equipment  losses 

South 

3 

Terrain 

Avenue  of  approach  restrictions 

South 

3 

Terrain 

Lines  of  communication 

North 

3 

Mission 

Accomplish  the  mission 

None 

3 

Enemy 

Enemy  equipment  losses 

South 

2 

Terrain 

Major  city  obstacles 

North 

2 

Own  troops 

Engineer  bridging 

South 

2 

Own  troops 

Force  positioning 

South 

2 

Own  troops 

Protect  flanks 

South 

2 

Terrain 

Dktftnrp  tn 

- - - —  --j— — 

Mona 

A  1VMV 

1 

L  * 

Indirect  Measures  of  Performance 

There  were  a  number  of  data  collection  instruments  used  to  collect  data  that 
contributed  to  the  interpretation  of  a  team’s  performance  on  the  experimental  task. 
Table  2  summarizes  the  types  of  data  collected  with  each  instrument  and  how  the  data 
were  used. 
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Indirect  data  collection  instruments,  data  collected,  and  purpose  of  the  data 


Table  2  (continued) 
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RESULTS 


The  results  are  presented  in  four  major  subsections.  The  first  provides  an 
overview  of  the  characteristics  of  the  participants.  The  second  subsection  describes 
results  for  task  performance  related  to  the  various  steps  or  phases  of  the  experimental 
task.  The  third  subsection  discusses  the  task  processes  in  more  detail.  And  finally, 
results  are  presented  on  participants’  responses  to  questions  about  the  tasks  and  the 
computer-support. 


Description  of  Participants 

A  summary  of  the  demographic  data  is  presented  in  Table  3.  There  was  a  total  of 
28  participants  in  the  experiment.  While  prerequisite  criteria  were  established  for 
participant^,  no  individual  was  excluded  from  the  experiment  because  of  failure  to  meet 
the  criteria.  Participants  were  not  dismissed  because  they  were  extremely  difficult  to 
obtain.  All  but  one  of  the  participants  met  at  least  three  of  the  four  selection  criteria  for 
the  experiment  (rank  of  major  or  higher,  combat  arms  officer,  branch  advanced  course 
graduate,  and  student  or  graduate  of  CGSOC).  Twenty-seven  participants  were  of  a 
rank  of  major  or  higher;  one  participant  was  a  captain  promotable. 

For  data  analysis  purposes,  teams  were  given  unique  designator  codes.  These  are 
used  in  the  first  column  of  Table  3.  The  first  letter  indicates  whether  the  team  was  in  an 
unspecified  (U),  structured  (S),  or  computer-supported  (C)  condition.  TTie  second  letter 
indicates  whether  the  first  COA  mentioned  in  die  Commander’s  Guidance  was  COA  N 
(A)  or  COA  S  (B).  The  numbers  indicate  the  sequence  in  which  teams  were  tested.  For 
example,  SA08  refers  to  a  structured  condition  with  COA  N  presented  first  and  the 
eighth  overall  team, 

'  Nineteen  of  the  28  participants  were  from  combat  (C)  or  combat  support  (CS) 
branches.  In  the  unspecified  condition,  five  were  from  C/'CS  branches  and  four  from 
combat  service  support  (CSS).  The  structured  and  computer-supported  conditions  each 
had  seven  C/CS  participants  and  three  from  CSS.  There  were  no  significant  differences 
among  the  groups’  task  and  tactical  knowledge  as  judged  by  the  experimenters  (Chi 
square  =2.77,  &=  25)  from  observation  of  performance. 

All  participants  were  graduates  of  an  advanced  course.  All  but  two  participants 
were  CGSOC  graduates.  The  two  non-CGSOC  graduates  (CA05,  see  Table  3)  had  just 
completed  the  first  trimester  of  the  course.  One  participant  was  a  graduate  of  the  Army 
War  College  and  another  was  a  graduate  of  the  Armed  Forces  Staff  College.  Three 
participants  completed  both  the  nonresident  and  resident  modes  of  CGSOC.  Three 
participants  completed  only  the  nonresident  mode  of  CGSOC. 

There  were  no  significant  differences  in  the  number  of  command  positions  held  by 
participants  among  experimental  conditions  (Kruskall-Wallis  statistics  1.25,  A =.54).  Nor 
were  there  any  differences  in  the  number  of  staff  positions  (Kruskall-Wallis 
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Table  3 

Participants'  background  characteristics 


Condition 

Rank 

Branch 

KEE9E5B9 

High 

Educ. 

Major 

hrEMkIBsII 

Poet  ions  in 

Cmd  |  Staff 

UA01 

LTC 

CSS 

mm 

22 

HA 

urban  Studies 

- 

79 

1 

0 

MAJ 

CSS 

D 

17 

MA/MS 

Int'l  Relat. 
Trans.  Mgt. 

- 

90 

0 

2 

UB04 

COL 

c/cs 

V 

2414 

BA 

Management 

1 

8 21-2 

3 

3 

LTC 

css 

1 

1814 

MS 

Logistic  Mgt. 

1 

.  «2 

1 

3 

UB09 

MAJ 

c/cs 

4 

15 

MA 

Pol.  Science 

84 

D 

1 

n 

CPT 

c/cs 

6* 

1114 

MA 

Construction 

83 

HI 

2 

1 

MAJ 

c/cs 

fra 

13 

ME 

Civil  Eng. 

84 

90 

2 

2 

H 

KAJ 

css 

wm 

17 

MA 

Business 

82 

90 

1 

3 

Averages: 

314 

1714 

1.4 

2.9 

SAQ3 

MAJ 

c/cs 

2 

13 

BS 

Engineering 

B 

85* 

3 

H 

MAJ 

c/cs 

414 

16 

BA 

• 

1 

89* 

2 

H 

SB  06 

MAJ 

css 

H 

1614 

■ 

Marketing 

- 

88 

D 

n 

KAJ 

css 

18 

Business 

81 

86 

1 

l_  3  ! 

SA08 

MAJ 

c/cs 

B 

14 

KEd 

Psychology 

83 

90 

i 

MW|: 

MAJ 

c/cs 

H 

17 

BA 

Secondary  Ed. 

- 

90 

0 

SB12 

MAJ 

c/cs 

1 

16 

JO 

Law 

- 

90 

3 

3  | 

MAJ 

c/cs 

3 

14 

MS 

EHSMMM 

83 

90 

1 

2  | 

SA13 

MAJ 

c/cs 

414 

14 

MA 

Engineering 

B 

90* 

2 

3  1 

MAJ 

css 

5 

16 

- 

Journalism 

HI 

90° 

1 

0  1 

Averages: 

4 

1514 

1.3 

3.1  | 

CB02 

LTC 

c/cs 

mm 

19 

MS 

Engineering 

■ 

85 

1 

B 

LTC 

c/cs 

m 

18 

MS 

Engineering 

HI 

84 

2 

HH 

CA05 

MAJ 

c/cs 

i 

1214 

MBA 

Finance 

n 

90* 

1 

MAJ 

c/cs 

i 

1214 

MS 

Info.  Systems 

90 

3 

3 

CA07 

MAJ 

css 

4 

15 

BS 

Art  Education 

89 

1 

mm 

MAJ 

css 

5 

16 

BS 

..  .... 

Social  Res. 

89 

1 

wm 

CB10 

MAJ 

c/cs 

15 

MA 

Military  Eng. 

90 

3 

mm 

MAJ 

c/cs 

14 

MBA 

Business 

90 

1 

HH 

Cb14 

MAJ 

c/cs 

5 

15 

MA 

■ 

90 

3 

MAJ 

css 

- 

14 

BS 

BBM 

HI 

90 

2 

.HH 

Averages:  214  15  1,8  4.0 


'Army  War  College  -  1990  Vlso  nonresident  CGSOC  -  1981 
s  A  need  Forces  Staff  College  ■  1984  sAlso  nonresident  CGSOC  -  1985 
Nonresident  course  eSelectee  for  SAMS 
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statistic  =1.50,  d  =  .47).  All  participants  had  command  or  staff  experience.  All  but  one 
participant  had  staff  experience  (an  average  of  3  assignments  per  participant).  Twenty- 
four  participants  had  held  command  positions  (an  average  of  1.6  positions  per 
participant).  Fifteen  of  the  twenty-four  participants’  commands  were  in  combat  arms  or 
combat  support.  Division  staff  experience  among  the  participants  included  an  assistant 
G3  and  DIVARTY  in  the  computer-supported  condition;  deputy  Gl/AG  in  the 
unspecified  condition;  and  G3  training,  Gl,  deputy  G4,  G3  plans,  and  EW/OPSEC  in 
the  structured  condition. 

Table  4  shows  responses  to  computer  experience  questionnaires.  There  were  no 
significant  differences  among  experimental  groups  on  whether  participants  had  taken 
computer  courses,  had  used  computers  with  a  cursor,  had  done  programming  for 
themselves  or  others,  or  computer  ownership  (Chi  Square  =  4.48,  ftc.25).  There  was  no 
difference  in  the  frequency  of  computer  usage  among  the  participants  of  the  three  groups 
(Kruskall  Wallis  statistic =3.66,  5: =.16). 

Requirements  were  relaxed  to  include  any  branch  when  participants  became 
available.  There  were  no  differences  on  indicators  that  cast  doubt  on  the  similarity  of 
the  groups. 


Task  Performance 


Courses  ofAEtion  ISkp  11 

Courses  of  action  (COAs)  were  provided  to  the  teams  through  the  Division 
Commander’s  guidance.  No  measures  were  taken  in  this  step. 


Pertinent  Facts ..(Step -2). 

Participants  became  familiar  with  the  scenario  and  gathered  pertinent  facts  using 
the  situation  and  reference  data  bases.  The  expert  panel  identified  a  total  of  24 
pertinent  facts  and  this  was  used  in  comparison  with  the  structured  and  computer- 
supported  teams. 

The  numbers  of  teams  recognizing  and  mentioning  facts  are  given  in  Table  5. 
Participants  in  the  unspecified  condition  were  not  required  to  list  pertinent  facts.  For 
comparison  to  the  structured  and  computer-supported  groups,  the  content  of  their 
discussions  was  analyzed  for  mention  of  any  of  the  pertinent  facts  previously  identified  by 
the  experts.  The  proportion  of  pertinent  facts  identified  did  not  differ  significantly 
among  the  experimental  groups.  Tire  groups  identified  an  average  of  only  22  percent  of 
the  pertinent  facts  identified  by  the  expert  panel.  (The  teams  in  the  structured  and 
computer-supported  conditions  received  the  expert  panel  list  of  facts  after  they  had 
completed  this  step.)  The  facts  which  were  not  identified  (narrowing  of  zone,  poor  cross 
country  movement,  required  loads  of  ammunition  and  POL  equal  basic  loads,  and 
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Table  4 

Participants'  computer  experience  (average  givers  for  “frequency  of  use,"  number  of 
yes's  given  for  Yes/No  questions). 


Teas 

Rank 

Type  of 
computer 
used?' 

Frequency 

of  use?2  - 

Taken  a 
computer 
course? 

Used  a 
mouse  or 
trackBel l? 

Programmed 

for 

yourself? 

Programmed 
for  some 
one  elae? 

Own  a 
computer? 

UA01 

LTC 

WS 

1 

No 

NO 

No 

No 

No 

MAJ 

US 

1 

Yes 

Yes 

Yes 

Yes 

Yes 

PC 

1 

US  04 

COL 

M 

5 

No 

No 

No 

No 

Yes 

PC 

5 

LTC 

WS 

5 

NO 

Yes 

Yes 

Yes 

Yes 

PC 

5 

U809 

MAJ 

PC 

5 

No 

Yes 

Yea 

NO 

mmt 

CPT 

PC 

5 

No 

Yes 

No 

No 

UA1 1 

MAJ 

PC 

3 

Yes 

Yes 

Yes 

No 

Yes 

MAJ 

• 

- 

No 

Yes 

NO 

No 

Yes 

M  -  Minico*putert/«ainf rmes  (DEC,  VAX,  IBM,  CDC.  etc.) 

US  -  Workstation  (Sui,  Apollo,  IBMST,  MASSCOHP,  Systolics,  etc.) 
PC  -  Personal  coasters  (IBM,  Macintosh,  Apple,  etc.) 


*  1  *  Less  than  tenthly  4  -  Every  few  days 

2  *  Monthly  5  -  Dally 

3  -  Weakly 
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adjacent  unit  attack  time)  were  important  but  not  neccessarily  critical  to  planning 
success. 

Five  of  the  expert-identified  facts  were  not  indicated  by  any  of  the  participants. 
The  facts  identified  by  the  most  teams  were  attack  mission  of  the  division,  strength  of 
the  opposing  enemy  division,  strength  of  own  division,  and  start  time  for  the  attack. 
These  facts  were  explicitly  referred  to  by  an  average  of  7  of  the  14  teams. 

Three  of  the  four  teams  selecting  the  Northern  COA  had  recognized  one  or  two 
of  the  facts  related  to  the  difficulties  with  the  Northern  COA  (shown  in  bold  type  in 
Table  5).  Team  SA13  identified  the  Fulda  as  unfordable  and  team  UA01  identified  the 
Haune  as  fordable  (though  both  teams  later  only  referred  to  the  requirement  for  a  river 
crossing  in  their  COA  justification).  Team  CB10  identified  both  facts.  None  of  the  four 
teams  selecting  the  Northern  COA  (contrasting  the  experts’  choice  of  the  Southern 
COA)  recognized  that  the  enemy  2nd  echelon  blocked  the  Northern  avenue  or  the 
limitations'  in  bridging  assets. 


List  Assumptions  (Step.  3) 

All  teams  were  given  a  set  of  assumptions  developed  by  the  expert  panel  to  help 
bound  excursions  they  might  be  tempted  to  take.  No  performance  measures  were 
collected  in  relation  to  this  step. 


Array,  the  fores.  (Steg-4) 


The  structured  and  computer-supported  teams  had  to  task  organize  friendly  and 
enemy  forces  for  the  main  attack,  the  supporting  attack,  and  reserve.  The  difference  in 
allocation  of  combat  power  relative  to  the  expert  panel  was  calculated  and  results  are 
presented  in  Table  6.  One  of  th 


re  computer-supported  tcam^  taiitu  iu  tuuipicic  me 


arraying  of  forces  partly  because  of  equipment  problems.  The  unspecified  teams  were 
not  explicitly  required  to  array  the  forces,  but  their  written  materials,  notes,  and  video 
tapes  were  inspected  to  determine  if  the  step  was  performed.  Two  unspecified  teams 
arrayed  the  forces  and  one  of  these  also  calculated  combat  power  ratios.  The  two  teams 
that  arrayed  forces  chose  the  non-preferred  COA  while  the  two  teams  who  failed  to 
array  forces  chose  the  COA  preferred  by  the  expert  panel. 


Positive  values  in  Table  6  for  main  and  supporting  attack  indicate  that  not  all 
combat  power  was  used.  Negative  values  for  reserves  indicate  that  a  higher  amount  of 
the  available  combat  power  was  left  in  reserve.  Inspection  of  the  Table  reveals  that  both 
structured  and  computer-supported  teams  allocated  noticeably  less  combat  to  the  main 
attack  than  did  the  expert  panel.  The  expert  panel  used  a  combat  power  of  26.5  in  the 
main  attack  (see  Figure  5).  The  differences  from  the  expert  panel  in  combat  power 
arrayed  are  shown  in  Figure  6.  The  structured  and  computer-supported  teams  provided 
26  and  45  percent  less  combat  power  to  the  main  attack,  respectively,  than  did  the 
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Table  6 

Mean  of  combat  power  differences  from  the  expert  panel  for  unspecified,  structured,  and 
computer-supported  teams 


Condition 

Statistic 

Main 

Attack 

Supporting 

Attack 

Reserves 

Array  Deficit 

Total 

Array 

Error 

I  Unspecified 

Observed 
(o  =  l) 

8.9 

05 

13 

10.7 

10.7 

Structured 

Mean  (n =5) 

6.8 

05 

..1.7 

5.6 

110  1 

Std  Dev, 

6.4 

1.9 

4.6 

8.2 

83 

Computer- 

Mean  (n=4) 

11.8 

1.0 

-0.7 

t  Hill  Hill—  ,,«>■  Li.l  ,, 

■ 

18.2 

supported 

Std  Dev. 

4.8 

3.3 

4.9 

1 

83 

Combined 

Mean 
(n  *  10) 

9.0 

0.7 

-1.0 

93 

143 

experts.  There  was  no  significant  difference  between  the  structured  and  computer- 

Cnmwt^H  tAQmc  in  VtA\u  tliAU  'utoifaH  /»AmKot  nAti/At*  frtr  «Yia  moin  r  /♦  fart 

«»v»f  MSVJ  Vwuik'Wk  VA  av»a  *nwm  l*bkMVn  y  fc  iVJl 

statistic  =  1.29,  &  *  .26). 


The  array  deficit  score  shown  in  Table  6  indicates  that  the  teams  failed  to  allocate 
all  their  assets.  The  total  combat  power  for  the  friendly  forces  was  44.9,  so  the  structured 
and  computer-supported  teams  failed  to  allocate  12  and  30  percent  of  the  available 
combat  power,  respectively.  The  team  following  unspecified  procedures  failed  to 
allocate  24  percent  of  the  available  combat  power.  Three  of  the  ten  teams  allocated  all 
of  their  combat  power  assets. 

? 

The  total  array  error  represents  the  absolute  difference  score  for  the  main  attack, 
supporting  attack,  plus  the  reserves.  Inspection  of  the  data  from  individual  teams 
indicates  that  a  large  portion  of  the  total  array  error  is  due  to  failure  to  allocate 
sufficient  combat  power  to  the  main  attack.  There  was  no  significant  difference  (t  test 
statistic  =  1.08,  & =,33)  between  the  structured  and  computer-supported  teams  in  their 
total  array  error. 


Critical  Events  .(Step  i) 

The  expert  solutions  for  COA  N  and  COA  S  contained  four  and  five  critical 
events,  respectively.  Each  COA  also  had  one  critical  event  for  the  supporting  attack. 
The  experts’  supporting  attacks  consisted  of  a  single  critical  event  (fixing  the  enemy)  for 
both  COAs  because  of  the  lack  of  strength  of  friendly  forces  combined  with  enemy 
disposition  and  environmental  considerations. 
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Figure  6.  Average  combat  power  arrayed. 


t*»»  cr  Were  CP 


Figure  6.  Average  difference  from  combat 
power  (CP)  arrayed  by  expert#. 
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Table  7 

Critical  events  correctly  identified  and  additional  ones  not  judged  critical  by  the  expert 
panel 


Critical  events  were  identified  by  the  structured  and  computer-supported  teams  for 
the  main  and  supporting  attacks  for  both  COAs.  One  of  the  four  unspecified  teams  also 
identified  critical  events.  A  correctly  identified  critical  event  required  a  match  to  the 
expert  panel’s  location  and  description.  One  computer-supported  team  did  not  array 
forces. 


The  structured  and  computer-supported  groups  identified  nearly  the  same  number 
of  critical  events  and  both  group  identified  the  majority  of  events  identified  by  the  expert 
panel.  Table  7  identifies  the  number  of  teams  in  each  condition  matching  the  experts’ 
critical  events,  the  average  number  of  correct  critical  events  for  teams  and  each  CO  A, 
and  the  number  of  extra  critical  events.  When  data  were  summarized  across  COAs  and 
types  of  attack  (main  and  supporting),  the  structured  and  computer-supported  groups 
identified  75  and  69  percent,  respectively,  of  the  critical  events  identified  by  the  expert 
panel.  Five  of  ten  teams  identified  at  least  80  percent  of  the  critical  events  identified  by 
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Table  8 

Averages,  standard  deviations  (in  parentheses),  and  sample  si2e  (n  =  )  of  raw  scores  for 
three  objective  war  gaming  factors 


Objective  Factors 

COA 

Experts 

Structured 

Computer- 

Supported 

Friendly 

North 

341 

130  (75) 
n=5 

104  (73) 

n«3 

Equipment 

Losses 

South 

327 

’  147  (53) 
n=5 

134  (110) 
n=3 

Enemy 

North 

219 

340  (197) 

n=3 

176  (157) 

n=3 

.Equipment 

Losses 

South 

214 

371  (181) 
n=3 

259  (260) 
n=3 

Battle  Duration 
(hours) 

— iMMwnirwH—a 

North 

47 

35  (19) 

n=5 

32  (14) 

n=2 

South 

33 

37  (19) 

n=5 

32  (14) 

n  =  2 

the  expert  panel.  All  teams  identifying  critical  events  identified  more  than  the  expert 
panel.  This  trend  was  more  pronounced  with  the  supporting  attack.  For  the  supporting 
attack,  direct  observation  of  performance  suggested  that  teams  believed  they  could 
continue  the  supporting  attack  past  the  first  critical  event.  The  expert  panel  considered 
this  unfeasible. 


The  unspecified  teams  were  not  required  to  identify  critical  events,  though  one 

'  I  hip  k  1/1  A<4  ♦»«  t/S  «MA4a1«1mA  AM  Am  /*V^  A  mm  J 

x  tiia  ivam  luvmmvu  iwu  iiuxiviutag  uiutai  cvcnui  uu  vavii  aiiu 

three  critical  events  which  the  expert  panel  did  not  recognize  as  critical.  This  team  also 
selected  the  Northern  COA. 
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War  Gaming  Critical  Events  (Step  6) 

It  was  up  to  the  individual  teams  to  decide  which  of  the  eight  objective  factors 
they  would  use  to  estimate  war  gaming  results.  At  their  own  choosing,  teams  could  also 
provide  separate  predictions  for  the  three  phases  of  a  critical  event  (preparatory,  during, 
consolidation)  or  a  single  prediction  for  each  critical  event. 

Table  8  presents  the  war  gaming  results  for  the  three  objective  factors  used  by  the 
expert  panel.  Figure  7  shows  the  differences  of  number  of  equipment  losses  and  battle 
duration  among  the  computer-supported  and  structured,  manual  groups  and  the  expert 
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Figure  7.  Average  objective  battle 
assessments. 


panel.  Variability  in  the  raw  data  was  high  for  all  three  measures.  Examination  of  data 
from  structured  and  computer-supported  teams  indicated  the  projections  of  more  friendly 
equipment  losses  for  COA  S  in  seven  of  eight  teams;  therefore,  projections  favored  COA 
N  for  this  factor.  These  predictions  were  contrary  to  the  expert  panel  who  projected 
more  losses  for  COA  N  (COA  N  =  341;  COA  S  =  327).  On  the  average  both  sets  of 
teams  under-estimated  the  friendly  equipment  losses  compared  to  the  expert  panel.  For 
enemy  equipment  losses,  the  predictions  of  all  seven  teams  from  both  groups  favored 
COA  S  as  did  the  predictions  of  the  expert  panel.  The  structured  teams  over-estimated 
the  enemy  equipment  losses  compared  to  the  expert  panel.  The  computer-supported 
teams  were  closer  to  the  expert  panel  estimates  on  this  factor. 

As  indicated,  the  predictions  of  the  expert  panel  for  the  two  equipment  loss 
measures  favored  one  COA  over  the  other  by  only  a  marginal  amount.  By  contrast,  the 
experts’  prediction  for  battle  duration  was  more  definitive;  the  values  were  47  and  33 
hours  for  COA  N  and  COA  S,  respectively.  Most  teams’  predictions  did  not  agree  with 
the  expert  panel’s  preference  for  COA  S.  The  duration  predictions  of  six  teams  favored 
COA  N  while  three  teams  favored  COA  S.  One  team  had  equal  battle  durations  for  the 
COAs. 


Scale  and  Aggregate  Battle  Kesuits  (Step  7  > 

Once  estimates  had  been  made  for  the  objective  factors  on  each  critical  event,  the 
teams  scaled  the  objective  factor  scores  using  a  9  point  scale,  to  allow  weighting, 
addition,  and  comparison.  A  low  scale  value  (*T)  means  a  low  utility  or  benefit  value. 
Figure  8  shows  pairs  of  scaled  scores  for  COA  N  and  COA  S  for  each  team.  The  top 
diagram  (Figure  8a)  shows  the  comparison  of  scaled  scores  for  friendly  equipment  losses. 
The  expert  panel  is  shown  in  the  middle  with  no  advantage  for  either  COA.  Three 
teams  had  an  advantage  going  to  COA  N,  another  team  had  the  advantage  to  COA  S, 
and  four  teams  had  no  difference. 

For  Enemy  Equipment  losses  the  middle  diagram  (Figure  8b)  shows  again  that  the 
expert  panel  gave  no  advantage  to  either  COA  Three  teams  had  an  advantage  going  to 
COA  S.  Three  other  teams  showed  no  difference  in  scaled  values. 

For  Battle  Duration  (Figure  8c)  the  expert  panel  gave  the  advantage  to  COA  S,  by 
two  scale  values.  One  team  from  each  of  the  two  experimental  conditions  agreed  with 
the  advantage  to  COA  S.  Four  teams  gave  the  advantage  to  COA  N  for  battle  duration 
and  three  teams  using  this  objective  factor  had  no  difference  in  scaled  values. 


Comparison  of  COA  (Step  81 

The  first  step  in  the  comparison  was  to  assess  the  COAs  on  subjective  factors. 
Each  structured  and  computer-supported  team  had  the  option  of  selecting  from  a  set  of 
four  subjective  factors  which  were  provided  on  the  work  sheets  or  to  add  factors  of  their 
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Figure  8a,  Friendly  equipment  ioee. 
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Figure  8b.  Enemy  equipment  lots. 
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Figure  8.  Scaled  battle  result  scores  for  COA  N  and  S 
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own.  Teams  also  assigned  a  scale  value  from  1  to  9  for  subjective  factors.  A  team  then 
weighted  the  individual  objective  and  subjective  factors  in  preparation  for  aggregation 
and  comparison  of  the  CO  As.  Based  on  the  aggregation  procedures,  the  COA  with  the 
higher  score  was  the  preferred  COA.  The  expert  panel  had  scores  that  favored  COA  S 
for  both  objective  and  subjective. factors.  Table  9  shows  the  raw  data  for  each  structured 
and  computer-supported  team  arid  the  expert  panel. 

Aggregate  scores  for  subjective  factors  for  the  two  COAs  were  noticeably 
different.  COA  S  was  favored  by  eight  teams  and  COA  N  was  favored  by  two  teams. 
Aggregate  scores  for  subjective  factors  indicated  the  COA  that  was  eventually  chosen  by 
all  ten  teams.  The  eight  teams  with  higher  subjective  scores  for  COA  S  chose  COA  S 
while  the  two  teams  with  higher  scores  for  COA  N  chose  that  COA. 


Not  all  objective  and  subjective  factors  were  considered  by  all  teams.  Table  10 
summarizes  the  number  of  teams  that  considered  a  particular  measure  and  the  average 
weighting  assigned  to  that  measure  when  it  was  considered.  Note  that  the  three  factors 
given  the  highest  weighting  by  the  participants  teams  were  some  of  the  highest  weighted 
factors  for  the  expert  panel.  Two  subjective  factors  (flexibility  and  risk)  that  were 
weighted  high  by  the  experts  were  not  rated  as  high  by  the  participants.  Examination  of 
the  data  indicates  considerable  inter-team  variability  for  both  of  these  factors  that  are 
very  difficult  to  predict,  particularly  risk. 


In  order  to  inspect  the  weighted  sums,  a  percentage  difference  was  computed  for 
each  teams’  pair  of  weighted  sums.  The  difference  between  objective  scores  for  COA  S 
and  COA  N  were  divided  by  each  team’s  combined  weighted  sum  on  the  higher  COA. 

A  similar  percentage  was  calculated  by  taking  the  difference  of  the  subjective  COA  S 
score  from  the  COA  N  score  and  dividing  by  the  total  weighted  sum  of  the  higher. 
Figure  9  shows  the  results.  The  darker  shading  shows  the  proportion  of  the  difference 
contributed  by  the  objective  factor,  and  the  lighter  shading  shows  the  subjective 
proportion.  This  figure  clearly  shows  the  predominance  of  the  subjective  factors  in 
distinguishing  between  the  COAs,  Three  teams  (SA03,  SB12,  and  CA07)  had  objective 
factor  results  which  supported  COA  N  but  a  greater  proportion  of  their  subjective  factor 
supporting  COA  S.  The  figure  shows  two  of  the  teams  (SA13  and  CB10)  who  chose  the 
non-expert  selected  COA.  Comparing  the  length  of  the  bars  in  the  figure  to  the  expert 
panel  value  also  shows  that  several  teams  selecting  the  expert-selected  COA  over¬ 
estimated  the  advantage  in  COA  S  by  a  factor  of  two  (in  the  case  of  CB02)  and  of  three 
(in  the  case  of  SB06). 


COA  comparison  errors.  The  critical  event  war  gaming  and  COA  summary 
comparison  sheets  of  the  structured  teams  were  checked  for  arithmetic  errors.  COAAT 
automatically  performed  arithmetic  operations  for  the  computer-supported  teams.  Four 
of  the  five  structured  teams  made  addition  or  multiplication  errors.  Team  SA08  made 
no  calculation  errors.  Team  SA03  added  columns  incorrectly  on  the  critical  event  war 
gaming  sheet.  For  POL  expended  on  COA  S  they  totalled  20%  instead  of  25%.  For 
ammunition  on  COA  S  they  had  50%,  instead  of  the  40%  they  would  have  had  if  their 
base  figures  had  been  added  correctly.  Team  SB06  added  duration  of  the  supporting 
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Objective  and  subjective  factor  war  gaming  results  by  team,  COA,  and  factors 


SA03 


Figure  9.  COA  S  -  COA  N  difference 
relative  to  total  score  of  preferred  COA 


Table  10 

Relative  weights  for  objective  and  subjective  factors  for  teams  and  the  expert  panel 


Factor  type 

Factor 

Number 
of  teams 

Average 

weight 

Experts’ 

weight 

Subjective 

Mission  accomplishment 

9 

95 

100 

Objective 

Battle  duration 

8 

90 

80 

Objective 

Friendly  equipment  losses 

\ 

8 

74 

100 

Subjective 

Effective  use  of  assets 

4 

73 

60 

Subjective 

Facilitate  future  opns 

9 

71 

40 

Objective 

Enemy  personnel  losses 

6 

70 

na1 

Objective 

FEBA  movement 

3 

70 

na1 

Objective 

Friendly  personnel  losses 

8 

69 

na1 

Objective 

Enemy  equipment  losses 

6 

68 

60 

Subjective 

Flexibility 

8 

60 

80 

Subjective 

Risk 

8 

53 

80 

Objective 

Ammunition  expended 

3 

33 

nau 

Objective 

POL  expended 

3 

27 

nau 

1  Not  weighted  by  the  expert  panel. 

2  Estimates  indicated  POL  and  ammunition  were  not  a  problem. 


attack  to  the  duration  of  the  main  attack.  Since  the  events  occur  simultaneously,  time 
should  not  be  added.  They  had  62  hours  for  the  duration  of  COA  S;  instead  then- 
figures  should  have  added  to  57  hours.  Team  SA13  also  made  an  error  in  battle 
duration  addition.  Their  figures  summed  to  24  hours  for  COA  N;  instead  it  should  have 
been  26  hours.  Team  SB  12  made  a  multiplication  error  on  their  COA  summary  work 
sheet.  A  weighted  scaled  score  for  ammunition  expended  on  COA  N  should  have  been 
33.0,  but  through  error  in  multiplication  they  had  34.8. 

Confidence  in  war  game  estimates.  In  the  questionnaire  phase  after  the 
experimental  tasks,  participants  from  the  structured  and  computer-supported  teams  were 
asked  to  rate  the  confidence  they  had  in  their  war  game  estimates.  Figure  10  shows  the 
average  confidence  ratings  by  objective  factors  and  teams.  Participants  gave  an  average 
rating  of  2  ("not  very  confident")  on  a  5  point  scale  over  all  eight  objective  war  game 
factors.  Computer-supported  teams  rated  their  confidence  of  enemy  equipment 
estimates  lowest  and  movement  of  FEBA  highest.  Structured  teams  rated  enemy 
personnel  loss  estimates  lowest  and  Class  Hi  (POL)  highest.  Only  once  was  an 
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"extremely  confident"  (5)  rating  made  for  any  of  the  factors,  while  19  percent  of  the 
individual  ratings  were  "not  at  all  confident"  (1). 

The  average  of  the  team’s  confidence  ratings  was  correlated  with  the  absolute 
value  of  the  difference  between  the  expert’s  estimates  (averaged  over  COA  N  and  S)  and 
the  team’s  estimates  for  three  war  gaming  factors  (friendly  equipment  loss,  enemy 
equipment  loss,  and  duration.  A  low  negative  correlation  (Spearman’s  correlation  =  - 
.1429,  &=.75)  was  obtained  for  friendly  equipment  loss  error  and  confidence.  (Note:  the 
participants  had  no  feedback  from  the  experts’  estimates  to  use  in  judging  confidence.) 
There  was  a  similar  relationship  between  the  error  in  duration  estimates  and  a  team’s 
confidence  in  that  estimate  (Spearman’s  correlation  =  -.2758,  d=.45). 

Teams  which  made  greater  errors  had  higher  levers  of  confidence  in  their  enemy 
equipment  loss  projections  (Spearman’s  correlation  =  .5268,  &  -  .2.4).  If  direction  of  the 
error  is  considered  (absolute  value  not  taken)  then  a  reversal  in  the  correlation  occurred. 
Teams  which  estimated  more  enemy  losses  than  the  experts  were  more  confident,  than 
those  which  had  lower  loss  estimates  than  the  experts  (Spearman’s  correlation  =  -.7946, 
fi=.03)  (see  Figure  11). 


COA  Decision  andJustificationJStep  9) 

COA  Selection.  The  expert  panel  preferred  COA  S.  This  COA  was  preferred  by 
two  of  four  teams  in  the  unspecified  condition  and  four  of  five  teams  for  both  the 
structured  and  computer-supported  conditions. 

Multi-attribute  utility  comparison  to  experts.  Table  11  shows  the  teams  who 
considered  justification  factors  that  were  used  by  the  expert  panel.  When  teams 
considered  a  factor,  they  usually  considered  it  in  the  same  context  as  the  expert  panel. 
One  exception  to  this  finding  was  that  battle  duration  was  incorrectly  assessed  as 

PAA  XT  U.t  1  A 

wri  isjr  vi^iu  \jl  at  ivauu* 


Teams  choosing  the  preferred  COA  had  a  mean  score  of  67  percent  (see  section 
2.5  for  scoring  rules),  while  teams  choosing  COA  N  had  a  mean  score  of  46  percent. 

Merit  of  justification.  Table  12  illustrates  the  scoring  logic  and  the  resulting  scores 
for  each  team.  Ties  in  the  point  totals  among  teams  were  broken  by  assessment  of  the 
quality  of  their  justifications  and  whether  they  recognized  the  importance  of  the  opposed 
river  crossing.  Table  13  provides  an  explanation  of  the  relative  quality  of  each  team’s 
justification  ranking. 

The  rank  orders  of  the  scores  were  significantly  different  showing  a  significant 
treatment  effect  (Kruskall-Wallis  test  statistic  =  5.99,  &-.05).  Both  the  computer- 
supported  and  structured  groups  performed  better  than  the  unspecified  group  (Mann- 
Whitney  test,  4 =.05).  There  was  not  a  difference  between  the  computer-supported  and 
structured  groups  (Mann-Whitney  test,  &  >  .05). 
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UB09 


Table  13 

Factors  used  in  scoring  and  ranking  team  COA  selection  and 
justifications. 


Rank 

Team 

C2A 

Factors 

Points 

1 

C802 

South 

•  opposed  river  crossing 

+4 

•  only  2  bridge  companies 

•  desire  to  defeat  18th  MTR  in  other  than  defensive  positions 

+3 

2 

CA07 

South 

«  18  TR  in  prepared  defensive  positions 

*4 

•  limited  engineering  and  bridges 

•  key  terrain  on  OBJ,  speed,  air  attacks 

+3 

3 

SA08 

South 

«  dug- in  HTR  in  2nd  echelon 

+4 

«  limited  bridge  assets 

+3 

•  defeat  IS  MRD  2nd  echelon  with  attack  helicopters  vs.  headlong  engagement 

4 

CA05 

South 

•  opposed  river  crossing 

♦4 

•  fordable  rivers  in  South 

♦2 

■m 

*  COA  N  has  more  time  risk,  better  chance  of  mission  success 

in  South 

5 

SA03 

South 

•  2nd  echelon-in  prepared  defense 

+4 

•  river  crossing 

•  greater  flexibility,  less  risk 

+1 

6 

SB06 

South 

•  frontal  attack  into  enemy 

♦4 

*  river  crossing 

•  least  movement,  positioning  of  units 

+1 

7 

SB12 

South 

•  Bridge  Fulda  [fordability  not  limited  assets) 

+2 

•  enemy  counterattack  at  Haune,  [possible  head-on  mtg  engag. 

•  shortest  time,  greater  flexibility 

•  does  not  allow  enemy  to  prepare  defenses 

•  if  support  attack  fails,  subject  to  flank  counterattack 

w/  reserve]  +2 

8 

UA11 

South 

•  15th  MRD  reserve  defends  forward  of  Fulda 

+2 

•  Fulda  fordable  in  South 

*  (bridging  an  advantage,  crossing  by  bridging  in  South) 

*2 

•  [fastest  avenue  to  objective  in  North,  main  attack  avoids  enemy  strength] 

9 

CB14 

South 

•  constrained  by  bridges 

•  sure  flexible 

♦3 

•  [faster  reaching  objective] 

10 

CB1Q 

North 

♦  river  crossing,  unfordable  river 

•  [bridge  assets  allow  crossing] 

*  [defeat  counterattack  disadvantage  in  South) 

♦  [North  faster] 

*2 

11 

SA13 

Ncrth 

•  river  crossing 

+1 

*  [may  encounter  elements  of  18  TR,  once  crossed  and  defeated  rapid  advance] 

•  [counterattack  by  18  TR  slows  South,  North  is  faster] 

12 

UA01 

North 

«  river  crossing 

•  [rivers  not  fordable  after  7  Sep] 

•  approach  to  rivers  favor  enemy  in  defense 

•  [time  advantage  in  North] 

♦1 

13 

US  04 

North 

•  none 

•  speed  advantage  in  South 

14 

UB09 

North 

•  none 

[terrain  more  advantageous  to  rapid  movement) 
[speed) 


Errors  are  placed  in  brackets  ([]).  Errors  and  quality  of  factors  were  used  for  breaking  ties  in  rank 
order. 
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Performance  Times 


Although  times  were  not  completely  free  to  vary,  measures  of  time  were  analyzed 
to  determine  any  differences  among  experimental  conditions  and  relationships  to  other 
variables.  The  total  time  to  complete  all  the  tasks  was  regulated  through  instructions. 

All  teams  were  given  a  completion  time.  Additional  time  was  granted  when  teams 
needed  it.  Structured  and  computer-supported  teams  were  also  given  completion  times 
for  each  step.  The  experimenter  reminded  participants  to  keep  within  time  requirements 
when  they  appeared  to  be  taking  too  long.  Teams  were  allowed  to  complete  tasks  faster 
than  the  allowable  time  for  each  step  and  were  allowed  to  go  on  to  the  next  step  if  they 
desired. 

Figure  12  shows  the  cumulative  average  times  (plus  and  minus  one  standard 
deviation)  for  structured  and  computer-supported  teams.  The  time  guidelines  are 
depicted  with  an  inverted  triangle  annotated  with  the  specific  time.  The  average  final 
time  for  the  unspecified  teams  is  shown  at  step  9  as  a  diamond  symbol.  The  graph 
shows  that  the  computer-supported  teams  tended  to  take  longer  than  either  the 
structured  teams  or  the  experimental  guidance  times  for  arraying  forces  and  identifying 
critical  events.  During  these  times  the  computer-supported  groups  were  inputting  data 
into  the  COAAT  decision  tool.  Their  relative  unfamiliarity  with  COAAT  (about  30 
minutes  of  training)  explains  the  longer  completion  times  for  these  steps.  There  were  no 
significant  differences  among  the  experimental  groups  on  the  total  time  to  complete  the 
tasks  (F[2,ll]=2.57,  <5  <  .20).  The  correlation  between  the  total  time  and  the  solution 
ranking  was  negative  (Spearman’s  correlation  =  -.35,  &  =  .22)  (faster  times,  poorer 
solutions),  but  not  significant.  Table  14  shows  that  the  critical  event  step  times  had  a 
significant  negative  correlation  with  solution  ranking  (faster  times,  poorer  solutions). 

Early  Decisions.  The  amount  of  time  used  to  make  a  decision  was  of  interest 
because  the  estimate  process  recommends  against  arriving  at  premature  conclusions;  that 
decisions  should  not  be  made  until  each  course  of  action  is  independently  analyzed  and 
those  results  compared.  A  data  collection  sheet  used  to  profile  each  team  was 
completed  by  experiment  observers.  The  observers  marked  on  these  sheets  when  it 
appeared  that  a  conclusion  had  been  made  prior  to  step  8,  compare  courses  of  action. 
Also  participants  were  asked  questions  if  a  conclusion  had  been  made  prior  to  step  8,  If 
one  had,  they  were  asked  whether  that  conclusion  was  the  same  as  the  final  choice  or 
not  and  the  earliest  step  in  which  they  made  a  decision  about  the  course  of  action. 

Figures  13  and  14  show  the  time  before  and  after  the  earliest  conclusion  of  either 
team  member  for  each  team.  If  the  teams  followed  the  sequence  of  steps  as  directed  by 
the  experiment  the  earliest  conclusion  would  not  be  reached  until  after  220  minutes  or 
85  percent  of  the  time  allotted  to  task  performance.  Two  teams  (SA13  and  CA05)  used 
a  higher  percentage  of  their  time  before  a  decision  was  first  reached  (94  percent  and  86 
percent,  respectively).  Four  teams  (SB12,  SA08,  CB02,  and  CB14)  used  less  than  50 
percent  (i.e.,  one  standard  deviation  less  than  the  85  percent  reference  point)  of  their 
time  before  they  came  to  a  conclusion.  There  was  no  significant  difference  among 
groups  in  terms  of  how  early  they  came  to  their  first  conclusion  (Kruskall- Wallis  statistic 
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Figure  12.  Average  cumulative  times  for 
task  steps  and  *1  standard  deviation. 


Table  14 

Correlations  between  task  step  times  and  solution  ranking. 


Steps 

Number  of 
Observations 

Spearman 

Correlation 

ft 

(2-sided) 

2  Gather  and  review  facts 

10 

-.37 

.30 

4  Array  forces 

10 

-.38 

.29 

5  Determine  critical  events 

10 

-.63 

.05 

6  War  game  critical  events 

10 

.36 

.31 

7  Aggregate/scale  results 

10 

.14 

.77 

8  Compare  COAs 

10 

.07 

.86 

9  Justify  COA  selection 

10 

-.20 

.59 

Average  cumulative  time  for 
structured  and  computer-supported 

10 

-.25 

.50 

- _ ~n  * - —  ~ 

1  A 

-nr 

V^LilllUldLl Y&  llllic  1U1  Si'.  ICcLUO 

=  0.39,  <5  =  .82).  Also  there  was  no  difference  of  earliest  decision  time  between  those 
teams  selecting  COA  N  and  COA  S  (Kruskall- Wallis  statistic  =  1.45,  a -.23).  There  was 
no  significant  correlation  between  a  team’s  justification  ranking  and  the  percentage  of 
elapsed  time  for  a  decision  (Spearman’s  correlation  =  .29,  ft =.16). 

At  least  one  member  in  each  of  the  ten  teams  following  structured  steps  came  to  a 
conclusion  before  comparison  of  COAs  (step  8).  Also  at  least  one  member  of  five  of 
these  teams  (SA03,  CB02,  CA05,  CB10,  CB14)  switched  conclusions  before  their  final 
decision.  Four  of  these  five  teams  switched  to  the  correct  COA  and  one  team  (CB10)  to 
COA  N. 

These  results  indicate  that  reaching  an  early  conclusion  does  not  impair  the  ability 
to  make  a  correct  conclusion.  Also  there  did  not  appear  to  be  any  primacy  bias; 
participants  changed  their  mind  for  the  better  in  4  out  of  5  cases. 
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One  of  the  objectives  of  analyzing  unspecified  procedures  was  to  see  what 
processes  were  followed,  the  variability  of  those  processes,  and  how  they  compare  to 
standard  processes.  The  unspecified  teams  were  guided  more  by  the  required  end  state 
of  their  assigned  task  than  were  the  structured  and  computer-supported  teams  or  than 
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Table  15 

Unspecified  teams’  process  sequence 


TEAMS 

UA01 

UB04 

UB09 

UA11 

Situation  assessment 

Situation  assessment 

Situation  assessment 

Situation 

assessment 

Comparison 

Comparison 

Array  forces, (combat  power  ratios) 

Comparison 

Concept  development 
pertaining  to  forces 
and  events 

Concept  development 
pertaining  to 
qualitative  forces 

Situation  assessment/  forecasting 

Concept 

development 

Visualization  of  COAs 

Comparison 

Check  completeness  against  mission 
goals 

Situation 
assessment 
(visualization  and 
forecasting) 

Check  mission  goals 

Situation  assessment 

Array  forces  (combat  power  ratios) 

Comparison 

War  gaming 

Comparison 

Comparison 

Selection  and 
justification 

Comparison  matrix 

Selection  and 
justification 

Concept  refinement 

Selection  and 
justification 

Array  forces  (combat  power  ratios) 

Comparison 

Array  forces  (combat  power  ratios) 

Comparison 

Selection  and  justification 

they  were  by  standard  estimate  procedures.  But  also  the  unspecified  teams  appeared  to  be 
less  certain  about  how  to  proceed  to  arrive  at  the  selection  and  justification  of  the  COA, 
Table  15  provides  a  general  summary  of  the  sequence  of  types  of  tasks  each  team 
performed. 

The  unspecified  teams  performed  in  very  different  styles  and  sequences.  One  of  the 
following  explanations  applies:  (a)  participants  did  not  know  the  process,  (b)  they  did  not 
know  how  to  apply  it,  (c)  they  thought  they  were  applying  it  but  weren’t,  or  (d)  they  did  not 
feel  it  was  appropriate  to  apply.  There  was  very  little  in  what  they  did  to  indicate  what  was 
the  end  of  one  process  step  and  what  began  the  next.  The  tasks  were  done  iteratively.  The 
four  unspecified  teams  all  did  comparisons  (strictly  qualitative  except  for  UB09)  between 
the  COAs  at  more  than  one  time.  Team  UB09  repeatedly  arrayed  forces  and  calculated 
combat  power  ratios.  These  ratios  were  only  for  the  force  postures  and  did  not 
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approximate  the  way  the  structured  and  computer-supported  teams  did  their  critical 
event  war  gaming.  With  the  exception  of  the  team  LJB09,  the  unspecified  teams 
performed  tasks  at  a  qualitative  level,  looking  for  features  to  discriminate  between  the 
COAs.  For  the  most  part  their  searches  were  done  in  an  unstructured  manner  viewing 
the  battlefield  as  a  static  situation. 

The  unspecified  teams  were  characterized  by  frequent  references  to  manuals, 
suggesting  their  desire  to  seek  more  guidance  on  the  task  process.  The  unspecified 
teams  used  an  average  of  2.25  of  four  manuals  (CGSC  ST  100-9,  FM  101-5,  FM  101-5-1, 
and  FM  101-10-1/2)  compared  to  1.4  and  1.5  average*  manuals  for  the  structured  and 
computer-supported  teams. 


Performance  PpKgss  Style 

Experimenters  observed  the  teams’  performance  for  characteristics  of  work 
management,  team  dynamics,  level  of  knowledge,  and  performance  style.  This  work  was 
tried  to  determine  if  there  were  any  differences  on  various  style  dimensions.  There  were 
two  general  reasons  that  these  dimensions  were  examined:  to  diagnose  differences 
among  the  three  experimental  groups  (see  Table  16)  and  differences  among  poorer  and 
better  solution  rankings  (see  Table  17)= 


Work  management.  Only  5  of  14  teams  made  a  conscious  effort  to  organize  and 
allocate  work.  Nine  of  14  teams  made  an  observable  effort  to  manage  their  time,  and  4 
teams  were  disciplined  in  controlling  and  avoiding  distractions.  There  were  no  real 
different  trends  among  the  three  experimental  groups.  Six  of  the  9  teams  who 
deliberately  managed  time  were  in  the  poorer  half  of  solution  scores. 


Team  dynamics.  Determining  roles  was  left  up  to  the  two  participants.  The  three 
teams  with  an  unbalanced  allocation  of  work  scored  in  the  poorer  half.  In  all  cases  a 
leader  and  subordinate  role  emerged,  but  some  limes  dominance  shifted  during  the  task 
session.  There  was  a  conflict  in  roles  in  2  of  the  14  teams.  The  leader  appeared 
adequate  in  his  role  of  leading  in  10  of  14  cases.  Again  there  were  no  real  differences 
among  the  three  experimental  groups.  The  teams  who  scored  in  the  better  half  with 
their  solution  justifications  all  had  adequate  leaders,  while  only  three  of  the  seven  teams 
in  the  poorer  half  had  adequate  leaders. 


Task  and  tactical  knowledge.  Only  one  of  the  four  (25%)  unspecified  teams  were 
judged  to  be  adequate  in  their  task  and  tactical  knowledge.  In  contrast  three  (60%) 
computer-supported  teams  were  judged  adequate,  and  four  (80%)  structured  teams  were 
adequate.  In  11  of  the  14  teams,  one  member  appeared  to  be  noticeably  more 
experienced,  knowledgeable,  or  more  motivated  than  the  other. 

Performance  style.  Five  teams  appeared  to  be  affected  by  "groupthink",  a  decision 
bias  where  there  is  unchallenged  and  unwarranted  agreement  in  an  attempt  to  avoid 
conflict.  Three  of  these  five  teams  selected  the  northern  COA.  Ten  of  the  14  teams 
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Table  16 


Team  Profiles:  Frequency  of  characteristics  by  experimental 
group 


Character i sties 

WorK  management - 

Consciously  organized  work 
Followed  work  allocation 
Referred  bock  to  plan 
Improved  on  plan 

Managed  tine  and  sequence  of  work 
Overtly  avoided  distractions 


Computer- 

Unspecified  Structured  sipported 

n=4  n=5  ri“S 


1  1  3 
1  1  3 
0  0  0 
1  0  1 
4  2  3 
0  2  2 


Team  Dynamics _ 

Balanced  team  effort 
Masters  shared  equsl  status 
Leader/stlx>rdinate  roles  arranged 
Subordinate  accepted  his  role 
Leader  was  at  least  adequate  in  role 


2  4  5 
0  0  3 
4  5  5 
4  4  5 
2  5  3 


Task  and  Tactical  Knowledge  _ _ _ _ 

Knowledge  was  adequate  1  4 

One  member  dominant  in  experience,  knowledge.,  or  4  5 

motivation 


Performance  Style  _ _ 

Evidence  of  "groupthink"  2  3 

Means  of  reaching  consensus  -  Discussion  3  2 

Means  of  resolving  conflicts 

Consent  to  subordinate  0  0 

Defer  to  superior  2  3 

Work  performed  just  adequately  3  4 

Practiced  error  checking  2  5 


3 

Z 


0 

5 

2 

0 

1 

3 
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Table  17 


Team  Profiles:  Frequency  of  characteristics  by  poorer  and  better 
solutions* 


Poorer  Half  Better  Naif 

Characteristics  rv*7  n*>7 


Consciously  organized  work 

2  » 

3 

followed  work  allocation 

2 

3 

Referred  back  to  plan 

0 

0 

Improved  on  plan 

1 

1 

Managed  tine  •end  sequence  of  work 

& 

3 

Overtly  avoided  distractions 

2 

2 

Balanced  team  effort 

3 

0 

Members  shared  equal  status 

1 

2 

Leader/subordinata  roles  srranged 

7 

7 

Subordinate  accepted  his  role 

7 

7 

Leader  was  st  least  adequate  in  role 

3 

7 

Knowledge  was  adequate 

3 

5 

One  mmber  dominant  in  experience,  knowledge,  or 

Motivation  ? 

5 

7 

Evidence  of  "groupthink" 

3 

2 

Means  of  reaching  consensus  -  Discussion 

5 

5 

Means  of  resolving  conflicts  - 

Consent  to  *t4»rdiruitc 

0 

2 

Defer  to  superior 

3 

2 

Work  performed  just  adequately 

5 

3 

Practiced  error  checking 

4 

6 

Better  half  «  teams  with  a  solution  ranking  of  1  through  7, 

Poorer  half 

*  team  with  rwtks  of  B  through 

14.  (Note  that  4  of  7  teams  in  poorer  half  selected  COA  N). 
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discussed  issues  to  reach  a  consensus.  When  conflicts  arose,  the  leader  of  two  of  the 
five  computer-supported  teams  tended  to  go  along  with  his  partner.  The  subordinate 
deferred  to  the  leader  in  five  of  the  nine  unspecified  and  structured  teams.  Eight  of  the 
14  teams  were  judged  to  be  performing  just  adequately  (a  motivational  dimension).  Ten 
of  14  teams  performed  error  checking,  six  in  the  better  half  and  four  in  the  poorer  half. 


Situation  Awareness  Test 


At  the  end  of  the  performance  phase  of  the  exercise  all  participants  were  given  a 
32  item  test  to  check  their  understanding  of  the  situation.  Multiple  choice  questions 
were  given  on  the  areas  of  mission,  enemy,  troops,  terrain,  and  time.  The  test  was 
developed  to  be  difficult  with  the  rationale  that  an  easy  test  would  not  distinguish  among 
individuals.  Figure  15  shows  that  the  scores  on  subsections  and  the  overall  test  were 
comparable  among  the  three  groups.  There  were  no  significant  differences  among  the 
three  conditions  (Chi  square  [8  degrees  of  freedom]  =  0.10,  &>.995).  The  average  score 
among  participants  was  54  percent  correct,  ranging  from  28  to  81  percent.  There  was  no 
significant  correlation  between  a  team’s  solution  justification  ranking  and  the  sum  of  the 
two  individuals’  situation  awareness  test  scores  (Spearman’s  correlation  =  -0.29,  5-  —  .31). 


By  considering  questions  answered  correctly  and  incorrectly,  one  can  infer  what 
information  was  best  acquired  and  what  was  not.  Table  18  lists  the  most  commonly 
known  and  missed  questions  in  the  situation  awareness  test.  Question  19  in  particular  is 
interesting  because  it  asks  where  the  Fulda  river  was  the  most  fordable.  (The  fordability 
of  the  Fulda  was  an  important  factor  in  the  COA  selection.)  Seventy-five  percent  of  the 
participants  answered  this  question  correctly.  Five  of  the  seven  participants  answering 
incorrectly  also  were  on  teams  scoring  in  the  poorer  half  of  the  solution  justification 
ranks.  Both  members  of  Teams  UB09  and  UB04  missed  this  question,  and  they  also 
ranked  last  and  second  to  last  (respectively).  Team  UB09  selected  COA  N.  Both 
members  of  each  of  the  three  other  team ,  selecting  COA  N  answered  this  question 


vun^tiijr. 


Of  the  four  teams  selecting  COA  N,  only  Team  CB10  (ranked  tenia)  correctly 


identified  the  Fulda  as  unfordable  in  their  solution  justification. 


I 

Indirect  Measures  and  Participant  Judgments 


Workload  Ratings 

All  participants  rated  workload  twice.  For  the  structured  and  computer-supported 
conditions,  workload  was  rated  after  gathering  of  the  facts  and  at  the  conclusion  of  the 
exercise.  During  the  second  rating,  participants  using  the  structured  approach  had  to 
provide  separate  ratings  for  Steps  4  through  9  of  the  experimental  task.  Although  the 
two  sets  of  data  are  divided  into  Step  2  and  Steps  4-9,  the  unspecified  teams  did  not 
have  steps  specified.  Participants  in  the  unspecified  condition  performed  one  set  of 
ratings  at  a  time  period  that  corresponded  to  the  end  of  the  "fact  gathering"  step  for  the 
other  two  conditions;  and  another  set  of  ratings  at  the  end  of  their  exercise.  For 


62 


Computer- supported  i^lSruotured  EZJ  UntptolHcd 


CM 

o 


63 
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Teet  Subsection*  (number  of  question*) 


purposes  of  discussion,  workload  will  be  categorized  as  low,  medium,  and  high  for  mean 
values  of  1  through  3,  4  through  7,  and  8  through  9,  respectively. 

Examination  of  workload  data  for  the  structured  and  computer-supported 
conditions  indicated  that  there  were  no  major  differences  in  ratings  across  Steps  4 
through  9.  For  this  reason,  ratings  for  these  steps  were  averaged  for  each  participant. 
This  procedure  also  made  it  possible  to  compare  two  sets  of  ratings  for  all  three 
treatment  conditions. 

Figure  16  reveals  that  workload  ratings  for  all  conditions  were  at  medium  levels 
except  for  physical  workload  which  was  rated  low.  There  were  no  major  differences  in 
workload  that  could  be  accounted  for  by  treatment  condition.  The  mean  ratings  were 
5.1  for  unspecified  teams,  5.5  for  structured  teams,  and  5.0  for  computer-supported 
teams.  Frustration  for  the  structured  condition  was  in  the  low-medium  range  for  Step  2 
and  increased  to  medium  levels  for  the  remainder  of  the  exercise. 


COA  Task  Evaluation  Questionnaire 

One  of  the  objectives  of  this  experiment  was  to  determine  which  task  steps  were 
performed  poorly  or  are  difficult  to  perform.  In  addition  to  the  objective  measures 
indicating  poor  performance  in  relationship  to  unspecified  and  structured  procedures  and 
compared  to  an  expert  panel,  participants  were  questioned  about  several  qualities  of  the 
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Figure  16.  Workload  ratings  by  workload  factor, 
experimental  condition,  and  task  step. 


task. 


One  quality  was  task  difficulty.  Figure  17  depicts  the  average  ratings  for  each  task 
step  and  each  experimental  condition.  Among  all  task  steps,  steps  6  and  7  were  rated 
the  highest  difficulty  (3.5  and  3.4  average,  respectively),  between  "some  difficulty"  (3)  and 
"considerable  difficulty"  (4).  Progressing  from  the  war  gaming  step  to  the  justification 
step,  difficulty  ratings  went  down  by  an  average  of  1.3  for  the  structured  and  computer- 
supported  teams,  while  it  only  decreased  by  0.3  for  the  unspecified  teams.  Difficulty 
ratings  by  the  unspecified  teams  for  step  9  were  0.8  points  higher  than  the  structured 
teams  and  1.1  points  higher  than  the  computer-supported  teams.  Having  done  the 
structured  analysis  perhaps  made  the  selection  and  justification  step  easier  for  the 
structured  and  computer-supported  teams. 

Confidence  ratings  of  war  game  factor  estimates  were  also  taken  (as  described 
previously, „see  page  47).  The  low  confidence  ratings  correspond  to  the  relatively  high 
difficulty  rating  for  war  gaming. 


Figure  18  shows  the  amount  of  effort  that  the  unspecified  teams  reported  for  each 
of  the  task  steps.  The  unspecified  teams  did  not  perform  each  task  step,  nor  to  the  level 
of  detail  of  the  other  two  groups.  This  figure  depicts  what  they  reported.  The  "octagon" 
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amount  of  effort  is  generally  inflated,  for  example  none  of  these  teams  aggregated  and 
scaled  war  game  results.  Principal  activities  which  were  observed  (see  Table  15) 
involved  assessment  and  review  (step  2),  compare  COAs  (step  8)  and  justify  COA 
selection  (step  9).  Team  UB09  was  observed  expending  consider  bie  effort  arraying 
forces,  but  they  did  not  explicitly  identify  critical  events,  war  game,  or  aggregate  and 
scale  war  game  results. 


Complex  ■Cognitive. Assessment  Battery  (CCAB) 

CCAB  is  an  experimental  battery  of  computerized  tests  which  are  theoretically 
related  to  C2  skill  requirements.  CCAB  was  used  to  acquire  baseline  ability  data  to 
relate  to  experimental  measures  of  performance.  Also  CCAB  was  used  to  help  control 
the  time  that  the  structured  and  unspecified  teams  were  exposed  to  the  experimental 
environment.  Structured  teams  were  given  five  of  CCAB’s  nine  subtests  for  one  hour, 
and  unspecified  teams  were  given  ail  nine  subtests  during  a  two  hour  period.  One  data 
set  was  obtained  from  15  participants  on  word  anagrams  (WA),  logical  relations  (LR), 
mark  numbers  (MN),  numbers  and  words  (NW),  and  information  purchase  (IP).  A 
second  data  set  was  obtained  from  six  participants  on  the  tower  puzzle  (TP),  following 
directions  (FD),  route  planning  (RP),  and  missing  items  (MI).  (Data  for  one  unspecified 
team  [UB04]  and  one  structured  team  member  [SA13]  were  lost  due  to  a  computer 
error.) 


For  an  average  of  the  five  subtests,  fourteen  of  the  15  participants  scored  within 
one  standard  deviation  of  norms  based  on  51  college  students  (Sprenger  &.  Fallesen, 
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Amount  of  difficulty 


17.  Difficulty  ratings  for  COA 
tasks  by  experimental  condition 


Table  19 

Spearman’s  correlations  and  significance  levels  of  CCaB  subtests  with  Solution 
Justification  Rankings  and  Situation  Awareness  Test  Scores 


Participants 
Taking  CCAB 
Subtests 

CCAB  subtests 

Situation  Awareness 
Test 

Correlation  4 

" 

Solution  Justification 

Con  elation  ft 

Word  anagram  (WA) 

r*  -  - 

.4438' 

.05 

.0046 

.87 

Unspecified  and 

Logical  relations  (LR) 

.1580 

58 

.2473 

38 

structured  (n  =  15) 

Mark  numbers  (MN) 

.1652 

56 

.5643 

.03 

Numbers  and  words  (NW) 

-.2065 

.46 

.1116 

.70 

Information  purchase  (IP) 

.1848 

.52 

.5509 

.03 

" 

Tower  puzzle  (TP) 

.6000 

23 

3857 

.48 

■  Unspecified 

Following  directions  (FD) 

.7714 

08 

-3286 

30 

(n=6) 

Route.  nlanninw  fRPt 

twv  r 

-.0714 

on 

./u 

Missing  items  (MI) 

.7714 

.08 

-3000 

39 

1989).  The  other  participant  scored  at  the  seventy-fifth  percentile  (assuming  a  normal 
distribution). 

Table  19  gives  CCAB  subtest  correlations  with  the  situation  awareness  test  scores 
and  with  solution  justification  rankings.  There  were  no  differences  on  the  five  subtest 
scores  between  the  unspecified  and  structured  teams,  indicating  no  ability  differences  in 
group  make-up.  Differences  on  MN  and  IP  subtests  did  approach  a  significant  level, 
(<S~.06)  with  the  unspecified  teams  having  higher  scores.  Both  the  MN  and  IP  subtests 
correlated  significantly  with  the  ranking  of  solution  justification  (Spearman’  correlation 
with  MN  =  0.56,  & =.02,  Spearman’s  correlation  with  IP  =  0.55,  6 -.03),  indicating 
poorer  CCAB  subtest  scores  with  better  rankings  (see  Figures  19  and  20). 

CCAB  subtest  scores  were  also  correlated  with  the  situation  awareness  test  scores. 
Route  planning  had  a  significant  correlation  with  the  situation  awareness  scores  (better 
on  CCAB-RP,  better  on  situation  awareness  test)  (see  Figure  21). 


Data  Importance 

After  the  exercise,  participants  were  asked  to  rate  the  importance  of  situation  and 
reference  data  elements.  For  situation  data  elements,  participants  selected  the  top  six 
oui  of  20  data  categories  and  ranked  those  six  in  order  of  importance.  The  rankings 
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FiQura  19.  Relationship  9(  solution 
ranking  and  CCAB'a  mark  numbers. 


Figura  20.  Relationship  of  solution 
ranking  and  CCAB'a  informal  ioi  purchase. 


Figura  21.  Relationship  of  situation 
awareness  and  CCAB  >outa  planning. 
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were  combined  across  participants  to  compute  a  weighted  average  used  to  rank  the  order 
of  importance  of  the  situation  data  category.  The  situation  data  category  and  their 
importance  rankings  are  given  in  Table  20.  These  rankings  were  compared  to  similar 
data  categories  from  an  experiment  by  Michel  and  Riedel  (1988).  There  were  no 
significant  differences  in  the  two  sets  of  rankings  (Mann  Whitney  z  statistic  =  0.2583, 
&=.80),  though  the  rankings  have  a  fairly  low  correlation  (Spearman’s  correlation  = 
.1391,  &=.60). 


Also  participants  ranked  the  importance  of  reference  data.  The  top  four 
categories  were  selected  from  13  elements  and  also  ranked  in  order  of  importance.  The 
corresponding  data  are  given  in  Table  21. 


Homan-Machine  Interface  Design 

The  Human-Machine  Interface  questionnaire  provided  feedback  from  the 
participants  on  (a)  ease  of  use  for  workstation  features  and  (b)  preferences  for  various 
map  features.  Ease  of  use  questions  for  one  set  of  questions  required  participants  to 
rate  a  feature  or  capability  from  1  to  5;  1  represented  "hard  to  use",  3  was  "OK",  and  5 
was  "easy  to  use". 

Gathering  and  identifying  the  facts  required  access  to  Situation  and  Reference 
data  bases  in  the  Tactical  Planning  Workstation.  The  ten  participants  from  the 
computer-supported  condition  gave  a  mean  rating  of  3.9  (range  3  to  5)  when  asked  how 
well  the  data  bases  were  labeled  and  organized  for  easy  and  logical  access.  Maps 
provided  critical  information  for  analyzing  tactical  course  of  action.  Participants  gave  a 
mean  rating  of  3.8  (range  2  to  5)  when  asked  whether  map  displays  were  laid  out  in  a 
logical  manner  so  that  they  could  find  the  information  quickly.  Several  respondents 
identified  problems  with  scrolling  maps.  Map  backgrounds,  scales,  and  map  features 
were  all  rated  an  average  of  4.  All  map  labels  were  unanimously  judged  as  consistent 
with  standard  Army  usage. 

A  low  rating  for  a  set  of  features  (e.g.,  map  scales)  indicates  the  most  useful  and 
the  highest  rating  indicates  the  least  useful.  The  map  scales  of  1:80,000  (mean  =  1.6  on 
a  4  point  scale)  and  1:160,000  (mean  =  1.6)  were  clearly  more  useful  compared  to  the 
1:400,000  (mean  =  3.1)  and  1:800,000  (mean  =  4)  scales.  Of  map  features,  grids  (mean 
=  2.4  on  a  6  point  scale)  were  marginally  preferred  over  roads  (mean  =  2.9),  built-up 
areas  (mean  =  3.0),  contours  (mean  =  3.0),  and  hydrography  (mean  =  4.2). 
Miscellaneous  features  were  least  useful  (mean  =  5.4). 

The  Tactical  Planning  Workstation  has  four  map  background  options:  vegetation 
(the  default  background),  shaded  relief,  elevation  banding,  and  null.  There  was  a  slight 
preference  for  the  shaded  relief  (mean  =  2.2)  and  elevation  banding  (mean  =  2.3)  over 
null  (mean  =  2,8)  and  vegetation  (mean  -  2.9)  backgrounds,  but  the  null  background 
was  selected  the  most  times  (4)  as  the  first  choice. 
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Table  20 

Data  importance  rankings 


Situation  Data 
category 

Staff 

Area 

Div  Cdr's  guidance 

G3 

Corps  FRAGO 

G3 

Weather  history 

G2 

Tactical  overlay 

G3 

Intelligence 

estimate 

G2 

OPFOR  composition 

G2 

Current  operations 

G3 

Task  organization 

G3 

OPFOR  artillery 

G2 

OPFOR  committed 

G2 

Participants  Weighted  Rank 

selecting  Average 

element 


25 


20 


2 


17 


21 


12 


8 


15 


1 


5 


OPFOR 

reinforcements 


G2 


Intelligence 

reports 


G2 


Personnel 

strengths 


G1 


Personnel  estimate 


G1 


8 


ClftKR  TTT  fPOTW 


Logistics  estimate 


a  a 


G4 


11 


Equipment  status 


G4 


Weather  forecast 


G2 


Class  V  (ammo) 


G4 


Losses  and  gains 


G1 


Tabie  21 

Reference  data  importance 


Reference  Data  Category 


-  Staff 


Area 


Number  participants 

selecting  data  Weighted  Rank 

element  in  top  4  Average 


- 

Organization 

G3 

20 

1.3 

■I 

Intell  composition 

G2 

18 

2.1 

mm 

Equipment 

G3 

11 

2.4 

3 

Strengths 

G2 

18 

2.7 

4 

Battle  losses 

G1 

2 

3.0 

5 

Strengths 

Gi 

17 

3.3 

7 

Intell  equipment 

G2 

7 

3.3 

7 

Equipment  characteristics 

G3 

4 

3.3 

7 

Supply 

G4 

5 

3.4 

9 

POW 

Gl 

0 

- 

Equipment  characteristics 

G2 

0 

- 

Minefields 

G3 

0 

- 

- 

Transportation  payload 

G4 

0 

- 

- 

To  conduct  the  experimental  task,  understanding  of  the  tactical  situation  required 
access  to  both  BLUE  and  OPFOR  unit  information  and  control  measures.  Participants 
rated  the  BLUE  UNIT  information  (mean  =  1,6)  the  most  useful.  A  moderate 
preference  was  expressed  for  OPFOR  Unit  information  (mean  =  2.1)  and  BLUE 
countermeasures  (mean  =  2.9). 

Ease  of  use  was  also  assessed  for  workstation  interface  using  a  rating  scale  of  1  to 
5  with  verbal  anchors  of  "Difficult  (l)"v  "Somewhat  Difficult  (2)",  "Somewhat  Easy  (3)", 
"Easy  (4)",  and  "Very  Easy  (5)."  With  this  scale,  COAAT  was  considered  to  be  between 
"somewhat  easy"  and  "easy"  to  use  for  assigning  critical  events  (mean  =  3.4)  and  easy  for 
inserting  and  aggregating  war  gaming  measures  (mean  -  4.0).  Participants  also 
considered  it  "easy”  to  "very  easy"  (mean  =  4.5)  to  coordinate  use  of  the  Sun  and 
Symbolics  workstations  while  identifying  critical  events. 

Participants  also  gave  mean  ratings  of  3.6  and  3.8  for  ease  of  use  of  "walking" 
menus  and  the  Task  Organization  Tool  (TOAST),  respectively.  These  ratings  are 
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between  the  "OK"  scale  value  of  3  and  the  "easy  to  use"  value  of  5. 


Summary  of  Debriefing  Questionnaires 

At  the  conclusion  of  each  exercise,  the  participants  were  asked  to  complete 
questionnaires  concerning  the  day’s  activities.  After  completing  the  questionnaires,  the 
participants  were  debriefed  by  the  experimenters  using  interview  questions  developed  in 
advance  for  each  of  the  three  conditions.  Table  22  provides  the  tabulation  of  responses 
to  questions  on  difficulties,  adequacy  of  time  and  information,  and  automation 
suggestions.  Table  23  provides  the  tabulation  of  responses  to  questions  unique  to  each 
team  in  the  three  conditions.  The  following  summarizes  the  highlights  of  participants’ 
responses. 

•  War  gaming  is  an  important  task,  but  it  is  time-consuming  and  complex. 

•H 

•  Locating  and  using  appropriate  reference  material  to  support  war  gaming  is 
difficult. 

•  A  majority  of  the  experiment  participants  indicated  a  low  level  of  confidence  in 
their  ability  to  war  game  and  visualize  the  battle. 

•  Simplified,  easy-to-access  checklists  would  tea  definite  help. 

•  Automation  would  assist  war  games  by  pro  ling  correlation  of  forces,  combat 
power  ratios,  time-distance  calculations,  attrition,  logistics  consumption  data  for 
selected  classes  of  supply,  digitized  terrain,  range  fans,  and  assorted  checklists. 

•  The  Tactical  Planning  Workstation  was  viewed  as  easy  to  learn  and  use  by  most 
participants.  The  major  area  of  difficulty  seemed  to  center  around  using 
computer-displayed  maps. 

•  A  majority  of  the  participants  would  have  liked  to  have  had  access  to  a  printer 
to  create  a  paper  environment  that  they  are  more  comfortable  with. 

•  Computer-supported  participants  were  quickly  able  to  learn  the  operation  of  the 
Tactical  Planning  Workstation  and  COAAT.  (It  should  be  noted  that,  even 
though  they  were  able  to  operate  the  system  effectively  for  the  specific  experiment, 
they  did  not  have  the  opportunity  to  develop  real  proficiency  and  exploit  many  of 
the  advantages  of  automation.) 
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Table  22 

Summary  of  responses  to  debriefing  questions  that  were  common  to 
unspecified  and  structured  teams. 


Unspecified 

Structured 

Question 

Response 

<r\*4) 

<n*5) 

What  amjor 

Uar  gaming  and  predicting. 

4 

difficulties 
did  you  have 

Becoming  familiar  with  the  problem. 

3 

in  COA 
analysis? 

Visualizing  the  battle. 

1 

FHs  do  not  provide  enough  guidance.  > 

 1. 

Time  was  adequate. 

4 

5 

Uas  there 
enough  tine  to 

But  it  was  constraining. 

2 

perfona  the 
task? 

Would  have  used  more  if  it  were  available. 

1 

■JL  — 

Would  have  liked  to  think  about  the  problem  over  night. 

1 

What 

Information  was  adequate. 

2 

additional 
information  is 

Here  logic  tics  data. 

1 

2 

necessary  to 
perform  the 

CLHTAG  mission  (  intent. 

1 

task? 

Something  to  capture  unit  locations  for  visualization. 

. 1 

Y  #*5  ■MfntfTttkf-  i  r*r*  tavui(<-3  hnun  lw»on  •  'L  . 

** 

T 

■J 

Force  ratios. 

3 

1 

Time  l  distance  calculations. 

3 

1 

For  what 
functions 

Planning  factors. 

3 

would 

automation  be 

Logistics  consumption. 

2 

useful? 

Attrition  rates. 

1 

Checklists. 

1 

Word  processing. 

1 

Range  fans. 

1 
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Table  23 

Debriefing  questions  unique  to  each  experimental  condition 


UNSPECIFIED 

Did  you 

No,  no  specific  plan  or  team  member  assignment. 

4 

develop  e  plan 

■ 

for  working  on 

Used  a  ''least  regrets"  method. 

1 

the  problem? 

Would  have  worked  better  with  a  plan. 

1 

Used  ST  100-9  es  a  guide. 

3 

Used  FH  101-5  for  format.  i 

1 

Tried  action-reaction  war  gaming  but  not  successful  (also  unsuccessful  in  CGliGC). 

1 

What  process 

Tried  tc  find  criteria  to  justify  selection,  tattle  iwasure*  were  too  abstract. 

1 

did  yen.1  use  to 
analyze  Cute? 

Reviewed  Corps  FRAGQ  and  Div.  Cdr's  guidance  for  factors  and  weighted  than. 

1 

Post  inportont  factor  was  enemy  disposition. 

3 

Most  important  factor  was. .pun  force's  ability  to  mess  and  meneuver. 

1 

Tried  visualizing  bottle,  but  had  too  little  experience  or  time  for'  only  1  CUA. 

2 

Calculated  time  requirements  in  relative  terms  (faster  -  slower).. 

2 

r  “  m'  1 — 

STRUCTURED 

War  gaming  is  difficult  and  tine  consuming. 

3 

Did  the  9  step 
process 

Lacked  confidence  in  predicting  outcomes. 

2 

provide  the 
structure 
necessary  to 

Task  was  more  difficult  than  in  CG50C  because  did  not  have  cs  many  officers  for 
support  and  did  not  have  3-4  days  to  work  the  prohibit. 

2 

perform  the 
task? 

Process  focuses  on  peace  tine  training,  way  not  bo  time  to  apply  it  in  war  time. 

1 

Process  should  have  reminders  of  the  battlefield  framework  and  operating  system. 

1 

Automation  tools  should  calculate  battle  outcomes. 

% 

Did  the  work 

Yes,  the  work  sheets  essisted  in  organizing  arid  analysing  dote. 

5 

sheets  assist 

fi.j  p^eafa.M.C^i 

Ur>rlr  ahoAfa  ijnr*  uostH  no  r-hylrl  ieva  _ 

your  analysis? 

Difficult  to  provide  bottle  outcomes;  the  Army  does  not  have  a  standard. 

2 

COMPUTER -SUPPORTED 

kem 

What  was  most 
difficult  in 

Sensitivity  cf  the  mouse  using  the  walking  minus. 

5 

operating  the 
Tacticel 

unfamiliar  with  computer-generated  naps. 

3 

Plamirg 

Holding  mouse  button  down  while  using  the  walking  menus. 

2 

Workstation? 

Nap  scroll  hers,  should  have  a  reference  system. 

1 

Provide  a  printer  to  provide  screen  duips  of  selected  infestation. 

5 

What  changes 

/.u tenet#  many  of  the  war  gaming  functions. 

1 

would  you 
racoMisend  to 

Install  COA.LT  on  the  Sun  workstation. 

1 

the 

Workstation? 

Put  wore  flexibility  into  the  editing  of  CGAAT. 

1 

Change  walking  menus  so  the  button  does  not  have  to  be  held  clown. 

1 
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SUMMARY  AND  DISCUSSION 


The  experimental  task  of  analyzing  tactical  courses  of  action  was  performed  under 
three  conditions.  Participant  teams  in  the  unspecified  condition  were  allowed  to  perform 
the  task  using  a  procedure  or  process  of  their  choosing.  This  formed  the  baseline 
condition  to  assess  potential  performance  enhancements  resulting  from  the  use  of  U.S. 
Army  prescribed  procedures  imposed  under  the  structured  and  computer-supported 
conditions.  Comparing  the  results  of  these  two  groups  also  addressed  whether  or  not 
computer  support  would  further  increase  the  likelihood  of  performance  enhancement 
when  a  procedural  approach  was  imposed  upon  decision  making. 

Teams  had  to  choose  between  two  alternative  courses  of  action.  The  preferred  or 
"benchmark"  decision  was  based  on  the  expert  panel  which  developed  solutions  for  each 
step  of  the  task. 


Limitations  on  the  Generalizability  of  the  Experiment 

Generalizability  is  the  degree  to  which  findings  from  this  experiment  apply  to  the 
broader  environment  which  was  represented. 

A  fairly  rigorous  experimental  paradigm  (for  tins  tactical  domain)  was  used  to 
ensure  standardization  in  scoring  and  for  comparing  manual,  structured  procedures  te 
computer-support.  For  example,  teair ;  in  a L  conditions  were  given  a  list  of  assumptions 
near  the  beginning  of  the  task.  Structured  and  computer-supported  teams  were  given 
standard  lists  of  facts  and  force  array  after  they  had  completed  those  steps.  While  this 
exihanced  the  ability  to  standardize  scoring  procedures  and  compare  between  the  teams, 
it  also  may  have  "leveled"  the  structured  and  computer-supported  teams’  information  and 
performance. 

The  small  sample  size  limited  the  statistical  power1  of  the  tests  and  complicated 
interpretation  of  the  results.  To  overcome  the  limitations  of  small  sample  sizes, 
considerable  effort  was  put  into  diagnosing  the  behaviors.  All  statistics  were  treated  as 
exploratory,  with  nonparametric  statistics  frequently  used,  and  the  actual  value  at  which 
the  null  hypothesis  v  ould  be  rejected  (i.e.,  &)  was  reported. 


Participants 

Experience  and  background  of  participants  for  the  three  conditions  were  similar. 
Three  or  four  of  the  participants  in  each  condition  were  from  the  combat  service  support 
branch,  the  other  participants  were  from  combat  and  combat  support  branches.  One  of 


'Statistical  power  is  the  likelihood  of  correctly  rejecting  a  hypothesis  when  it  is  actually  false. 
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the  four  unspecified  teams  knowledge  was  judged  adequate  compared  to  four  for 
structured  and  three  for  computer-support.  Five  of  the  seven  teams  ranking  in  the  upper 
half  of  solutions  were  judged  to  have  adequate  knowledge,  and  three  of  seven  in  the 
poorer  half  were.  The  difference  is  not  significant  and  is  even  more  interesting  when 
one  considers  that  the  second  best  team  was  judged  not  to  have  adequate  knowledge, 
while  the  last  place  team  was  judged  to  have  adequate  knowledge.  There  was  no  trend 
between  the  quality  of  solution  and  the  combat/combat  support  and  combat  service 
support  make-up  of  the  teams. 


Task  Performance 

Analysis  of  results  was  done  on  the  decision  making  process.  The  following 
summarizes  the  results  as  they  pertain  to  the  procedural  steps  used  during  decision 
making. 

•M 

Pertinent  Facts  (Step  21 

According  to  training  on  the  estimate,  a  preliminary  analysis  should  be  conducted 
to  identify  the  pertinent  facts.  For  the  experiment,  there  were  no  differences  in  the 
average  number  of  facts  identified  across  conditions.  Teams  provided  a  surprisingly  low 
percentage  of  the  expert-identified  facts.  There  was  not  a  single  fact  that  was  listed  in 
common  by  every  team.  None  of  the  facts  were  listed  by  all  teams.  There  were  only  a 
few  facts  that  were  listed  by  the  majority  of  the  teams.  Participants  may  have  had 
insufficient  practice  performing  this  step,  there  may  have  been  uncertainty  as  to  what 
constituted  a  pertinent  fact,  or  individual  differences  in  what  is  considered  important  may 
be  the  most  dominant  effect.  Explicit  identification  of  facts  is  seldom  seen  in  command 
post  exercises.  Facts  were  sometimes  mentioned  later  in  the  experiment  after  the  step 
had  been  completed.  While  teams  were  supposed  to  record  and  update  pertinent  facts 
throughout  the  task,  they  never  returned  to  the  formal  step  of  recording  additional  or 
chanced  nertinent  facts. 

V  4 


Arraying  thg-Forres  (Step  4) 

Friendly  forces  must  be  arrayed  and  task  organized  to  ensure  an  acceptable 
probability  of  mission  success  based  on  the  commander’s  guidance.  Only  one  of  the 
unspecified  teams  arrayed  forces  and  calculated  combat  powers  and  ratios.  AH  teams  in 
the  structured  and  computer-sup*  irted  conditions  allocated  lsss  combat  power  to  the 
main  attack  than  the  experts.  Failure  to  allocate  all  of  their  available  combat  power  was 
often  due  to  inadequate  use  of  aviation  assets  and  field  artillery.  None  of  the  treatment 
manipulations  "guaranteed"  adequate  consideration  of  combat  resources.  There  were  not 
explicit  instructions  telling  them  to  use  all  available  combat  power,  nor  did  COAAT 
indicate  anything  about  available  or  remaining  combat  power.  In  the  future,  procedures 
or  computer  support  should  be  specifically  designed  to  address  the  problem. 
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Critical  Events  (Sleo31 


Critical  events  are  those  explicit  and  implicit  tasks  that  are  paramount  for  mission 
success.  There  was  no  difference  between  the  structured  and  computer-supported 
conditions  in  the  number  of  critical  events  they  identified  in  common  with  the  expert 
panel.  The  teams  identified  critical  events,  accurately  matching  a  high  number  of  the 
expert-identified  critical  events.  However,  the  teams  also  identified  noticeably  more 
critical  events  than  the  expert  panel,  particularly  for  the  supporting  attack.  Observations 
suggest  that  the  teams  were  overly  optimistic  about  the  extent  to  which  follow-on  actions 
were  feasible  after  the  first  critical  event  of  the  supporting  attack.  Only  one  unspecified 
team  identified  critical  events,  and  they  did  not  use  them  subsequently  to  war  game. 


War  Gamine  (Step  6\ 

War  gaming  required  teams  (using  a  prescribed  procedure)  to  make  quantitative 
predictions  for  objective  and  subjective  factors.  Eight  objective  or  battle  outcome  factors 
were  presented  to  participants  along  with  five  subjective  factors.  For  a  particular  factor, 
COA  comparison  focused  on  the  relative  values  for  the  two  courses  of  action  to  identify 
the  favored  COA  for  that  factor.  Aggregate  scores  for  both  objective  and  subjective 
factors  were  also  calculated  by  the  structured  and  computer-supported  teams.  None  of 
the  unspecified  teams  war  gamed  quantitatively. 


Of  the  objective  factors,  the  expert  panel  considered  friendly  equipment  losses, 
enemy  equipment  losses,  and  battle  duration  as  the  most  important.  There  were  no 
statistical  differences  between  the  structured  and  computer-supported  teams’  estimates  or 
differences  from  the  experts  on  these  three  objective  factors.  The  variability  in  the 
teams’  estimates  was  high,  which  is  another  indication  of  the  iow  confidence  that  the 
participants  reported  in  their  estimates.  Both  groups  underestimated  friendly  losses  for 
both  COAs.  The  structured  group  overestimated  enemy  losses  for  both  COAs.  All 


seven  teams  using  battle  duration  underestimated  the  duration  of  COA  N. 


The  teams  did  differ  from  the  experts  in  the  direction  of  the  advantage  between 
the  COAs.  Seven  of  eight  teams  favored  COA  N  on  friendly  losses,  while  the  experts 
gave  the  advantage  to  COA  S.  Six  of  six  teams  favored  COA  S  on  enemy  losses 
agreeing  with  the  experts.  Only  three  of  seven  teams  agreed  with  the  experts  in  favoring 
COA  S  on  battle  duration. 


These  results  suggest  notable  difficulties  of  participants  to  accurately  predict  time 
and  space  relationships  of  the  battle. 


Scale  and  Aggregate  Battle  Results  (StepJ71 

The  objective  and  subjective  factors  that  were  weighted  heaviest  by  the  expert 
panel  were  generally  weighted  high  by  the  participant  teams.  Experts  considered  two 
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objective  (battle  duration  and  friendly  equipment  casualties)  and  two  subjective  factors 
(mission  accomplishment  and  flexibility)  as  being  most  critical.  Participants  weighted 
two  of  these  factors  (mission  accomplishment  and  battle  duration)  very  comparably  with 
the  expert  panel;  however,  two  factors  (friendly  losses  and  risk)  were  weighted  lower  by 
the  participants  than  by  the  panel. 

Generally,  scale  values  did  not  strongly  favor  one  COA  over  the  other  for  the 
objective  me  ares.  The  experts  did  not  have  a  scale  value  difference  for  enemy  or 
friendly  equipment  losses;  only  battle  duration.  A  team’s  scale  values  for  the  two  COAs 
rarely  differed  by  more  than  one.  ( 


Comparison  of  CQA  (Step  8) 

The  percentage  difference  between  the  objective  scores  (weighted  and  summed) 
for  the  two  COAs  was  typically  small  (less  than  6  percent).  This  was  not  true  for 
subjective  factors.  Projections  for  a  particular  factor  by  teams  were  often  noticeably 
different  (at  least  greater  than  18  percent)  for  the  two  COAs.  The  subjective  factors  for 
this  scenario  provided  the  best  discriminators  for  selecting  a  COA.  Three  of  the  teams 
went  against  the  direction  of  their  objective  factor  advantage  and  made  their  selection, 
presumably,  based  on  the  subjective  factor  difference  between  the  COAs.  In  fact,  the 
team’s  subjective  score  favoring  a  COA  corresponded  perfectly  with  the  COA  which  was 
eventually  selected  by  that  team. 

All  of  the  structured  teams  were  observed  to  perform  error-checking  at  some  time 
during  the  overall  task,  showing  that  they  were  aware  that  it  should  be  done;  but  most  of 
them  experienced  arithmetic  errors  in  their  weighting  or  addition.  Hone  of  the 
computer-supported  teams  made  comparable  errors  because  of  the  spreadsheet  tool 
(COAAT)  available  to  do  the  calculations. 


COA  Decision  and  Justification  (Step  91 

Teams  under  all  experimental  conditions  were  required  to  choose  a  preferred 
course  of  action  and  to  provide  justification  for  the  decision.  For  purposes  of  analysis, 
information  for  the  justification  was  obtained  from  the  team’s  list  of  advantages  and 
disadvantages  for  each  course  of  action  and  from  a  paragraph  that  was  written  to  justify 
the  decision. 

The  groups  differed  significantly  in  the  merit  of  their  justification  for  selection  of 
COAs.  Computer-supported  and  structured  teams  ranked  significantly  higher  than  the 
unspecified  teams.  Three  teams  (2  computer-supported  and  1  structured)  reached  the 
maximum  possible  score  for  the  merit  justification  measure.  Two  unspecified  teams 
received  no  points  at  all.  The  four  teams  selecting  COA  N  ranked  tenth,  eleventh, 
twelfth,  and  fourteenth. 
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The  computer-supported  and  structured  teams  received  more  procedural  guidance 
than  the  unspecified  teams.  The  unspecified  teams  did  not  follow  the  same  procedures 
as  required  of  the  other  two  groups  (and  included  in  the  estimate  of  the  situation).  Only 
one  of  the  four  unspecified  teams  arrayed  forces  (quantitatively),  and  only  one  team 
identified  critical  events.  None  of  the  unspecified  teams  performed  war  gaming  of 
critical  events,  and  they  did  not  scale  or  aggregate  battle  results. 

As  it  turned  out  the  estimation  of  battle  results  by  structured  and  computer- 
supported  teams  distinguished  very  little  between  the  two  COAs.  However,  by  having  to 
make  these  estimates  for  critical  events,  the  structured  and  computer-supported  teams 
were  forced  to  visualize  a  level  of  the  dynamics  of  the  battle  that  the  unspecified  teams 
did  not  do  explicitly.  Although  many  times  the  structured  and  computer-supported  teams 
were  making  unsophisticated  guesses  to  satisfy  the  spreadsheet  with  a  number,  they  did 
address  each  critical  event  in  sequence.  There  was  very  little  projection  or  visualization 
by  the  unspecified  teams.  They  made  comparisons  on  the  face  value  of  the  information 
without  considering  the  dynamics  and  interaction  of  battle  events  and  battlefield  systems* 


Performance  Times 

The  time  to  complete  the  experimental  task  was  controlled  with  guidance  in 
instructions  and  while  monitoring  the  performance.  The  intent  was  to  let  the 
participants  control  the  time  spent  on  each  step,  but  total  participant  control  would  have 
risked  uncompleted  tasks.  Although  not  entirely  free  to  vary,  time  was  analyzed  as  a 
dependent  variable. 

There  were  no  significant  differences  among  the  experimental  conditions,  but  the 
computer-supported  teams  tended  to  take  longer  on  the  average.  There  was  also  a  trend 
(non-significant)  for  faster  times  to  lead  to  poorer  solutions.  This  was  especially  true  for 
the  critical  event  step  times  which  had  a  significant  negative  correlation  with  solution 
ranking.  This  might  be  because  spending  more  time  considering  the  critical  events 
provides  a  better  understanding  or  an  indication  of  a  greater  desire  to  understand  the 
future  action.  An  alternative  explanation  is  that  it  took  longer  to  enter  the  critical 
events  into  COAAT  and  that  the  teams  using  COA-VT  did  better  overall. 

A  special  case  of  time  was  considered.  Participants  were  asked  and  were 
observed  to  see  if  they  made  a  COA  determination  prior  to  the  formal  selection  step, 
referred  to  in  the  standard  training  materials.  The  experimental  guidelines  allocated  85 
percent  of  the  task  time  prior  to  when  a  selection  was  to  be  made;  while  the  remaining 
15  percent  was  for  justifying  the  selection.  All  but  one  team  used  less  than  85  percent  of 
their  time  before  a  selection  was  made.  Four  teams  used  less  than  50  percent  of  their 
total  time.  No  penalty  resulted  to  those  teams  making  early  decisions.  Also  there  was 
no  primacy  (or  "stubbornness"  or  'evidence  marshalling")  bias,  as  five  "early  decision" 
teams  eventually  switched  their  selection. 
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Other  Performance  Indicators 


Performance -Styk 


The  teams  were  observed  to  have  poor  task  organization  skills.  Nine  of  the  teams 
made  no  conscious  effort  to  organize  and  allocate  work.  This  is  especially  surprising 
because,  even  though  members  on  some  teams  were  previously  acquainted  with  one 
another,  none  of  them  performed  this  type  of  work  together  before.  Six  of  the  nine 
failing  to  organize  were  in  the  poorer  half  of  the  solution  rankings.  Nine  of  the  teams 
did  not  perform  ongoing  management  of  their  work.  , 


None  of  the  computer-supported  teams  exhibited  behaviors  normally  associated 
with  the  phenomenon  of  "groupthink",  while  five  of  the  nine  unspecified  and  structured 
teams  did.  The  computer-supported  teams  tended  to  have  more  of  the  qualities  (i.e., 
balanced  team  effort,  equal  status  on  team,  subordinate  acceptance  of  role,  one  member 
pot  dominant  in  knowledge  or  motivation)  associated  with  cohesion  and  cooperation 
than  the  other  teams.  This  points  to  the  importance  of  cooperation  and  consensus¬ 
building,  in  contrast  to  adhering  to  strict  hierarchies  or  agreeing  just  to  avoid  conflicts. 


In  all  cases  a  leader-subordinate  relationship  emerged.  In  at  least  one  team  the 


roles  switched  as  one  member  took  charge  from  the  otht 
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the  solutions  were  all  judged  adequate  by  the  observing  experimenters,  while  only  3  in 
the  poorer  half  were  judged  adequate. 


Situation.  Awareness 

There  were  no  significant  differences  among  experimental  groups  in  participants’ 
situation  awareness  accuracy.  The  participants  scored  an  average  of  only  54  percent 
correct  on  this  multiple  choice  test.  The  most  interesting  question  pertaining  to  the 
scoring  of  solutions  concerned  the  tordabiiity  of  the  Fulda  river.  Five  of  seven 
participants  answering  incorrectly  were  on  the  poorer  scoring  teams. 


Workload  .Eatings 

Participants  were  required  to  provide  their  perceptions  of  workload  imposed  by 
the  experimental  task  and  conditions.  Workload  was  partitioned  into  the  areas  of  mental 
demand,  physical  demand,  temporal  demand,  performance,  effort,  and  frustration. 
Workload  for  these  six  dimensions  was  categorized  as  being  either  low,  medium,  or  high. 

Workload  assessments  were  critical  for  addressing  several  issues.  In  general, 
workload  was  assessed  to  determine  if  using  a  structured  approach  to  decision  making 
increases  workload.  It  was  also  predicted  that  use  of  computer  support  would  reduce 
workload  imposed  by  use  of  a  structured  approach  but,  from  a  practical  perspective, 
there  was  concern  that  use  of  computer  support  could  actually  increase  workload.  It  is 
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not  uncommon  for  a  computer  system  to  shift  the  user’s  requirements  from  calculations 
and  other  reasoning  skills  to  lower  level  data  entry  skills.  The  shift  may  result  in 
increasing  workload.  If  the  computer  does  not  "fit"  well  with  how  the  users  perceive 
their  jobs,  the  mismatch  can  cause  an  increase  in  workload  as  the  user  might  try  to 
perform  their  tasks  both  with  computer-support  and  manually. 

There  were  no  significant  workload  differences  among  the  experimental 
conditions.  Workload  levels  were  at  medium  levels  for  all  areas,  except  for  physical 
demand  which  had  a  low  level.  The  small  sample  size  is  an  obvious  constraint,  but 
generally,  there  was  no  indication  (a)  that  use  of  a  procedure  imposed  more  workload  on 
the  participants  and  (b)  that  computer  support  made  the  job  easier.  While  computer 
capabilities  did  not  decrease  worldoad,  it  is  also  encouraging  that  workload  was  not 
increased,  especially  since  the  level  of  training  was  relatively  low  and  practice  was  nil. 
Only  two  hours  of  training  and  experience  were  provided  in  using  and  understanding  the 
computer-based  systems. 


Course  of  Action  Task  Evaluation 

Participants  were  asked  to  rate  and  comment  on  various  characteristics  of  the  task 
steps.  War  gaming  had  the  highest  average  difficulty  rating,  which  is  corroborated  by  the 
fact  that  the  unspecified  teams  did  not  even  perform  it.  The  perceived  difficulty  of  the 
selection  and  justification  step  was  lower  for  the  structured  and  computer-supported 
teams  than  for  the  unspecified  teams.  This  suggests  that  having  done  war  gaming, 
though  it  is  difficult  to  do,  makes  the  selection  and  justification  easier. 


Complex  Cognitive  Assessment  Battery  (CCAB) 


There  were  no  differences  between  the  unspecified  and  structured  participants 
taking  the  same  five  CCAB  subtests.  The  "Mark  Numbers"  and  "Information  Purchase" 
subtests  were  the  only  ones  having  a  significant  correlation  with  soiution  ranking.  The 
correlation  was  in  the  opposite  direction  from  what  was  expected:  poorer  CCAB 
performance  led  to  better  rankings.  "Route  planning"  had  a  significant  positive 
correlation  with  the  situation  awareness  test.  These  subtests  should  be  studied  further  to 
determine  if  they  indeed  are  testing  skills  or  abilities  which  correspond  to  performance, 
or  if  the  significant  correlations  are  happening  by  chance. 


Data  Importance 

There  were  two  broad  classifications  of  tactical  data  available  in  this  experiment, 
One  was  reference  data,  indicating  typical  base  information  which  a  planner  could  use  in 
force  projections,  war  gaming,  or  better  understanding  of  possible  force  structures  and 
capabilities.  The  five  top-ranked  reference  categories  were  organizations,  intelligence, 
own  equipment,  enemy  strengths,  and  own  personnel  battle  loss  percentages.  Situation 
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data  were  made  up  of  information  about  the  tactical  events  which  occur  and  their 
resulting  effects  on  force  status.  The  five  top-  ranked  situation  data  categories  were 
division  commander’s  guidance,  Corps  FRAGO,  weather  history,  tactical  overlay,  and  the 
intelligence  estimate. 


Human-Machine  Interface  Design 

A  questionnaire  was  used  to  obtain  opinions  from  participants  on  human  factors 
considerations  related  to  computer  support.  Despite  limited  training  with  the  computer 
systems,  participants  indicated  no  significant  problems  associated  with  accessing  data 
bases,  and  generally,  the  computer  support  provided  by  the  COAAT  was  considered 
"somewhat  easy"  to  use. 


The  questionnaire  also  addressed  user  preferences  for  the  utility  of  certain 
features  of  the  computer-support  systems.  The  Tactical  Planning  Workstation  had  four 
map  scale  options,  and  generally,  participants  preferred  to  work  with  the  two  larger  scale 
maps  (1:80,000  and  1:160:000).  For  the  map  features,  participants  found  map  grids 
slightly  more  useful  than  roads,  built-up  areas,  and  contours  while  miscellaneous  features 
were  not  very  useful.  Participants  also  showed  a  slight  preference  for  the  shaded-relief 
and  elevation-banding  map  backgrounds,  but  the  null  background  was  most  often  picked 
as  the  first  choice.  More  frequent  use  of  null  background  was  aiso  found  by  Michel  and 
Riedel  (1988).  Most  teams  indicated  that  the  mouse  was  too  sensitive,  i.e.,  it  required 
exceptionally  fine  motor  control  to  perform  distinct  operations  (especially  on  the  walking 
menus). 


Eartfclpant  CommtflteJPiiPng  fietalefing 

Each  team  was  asked  a  set  of  questions  at  the  conclusion  of  the 

experiment.  The  following  summarizes  comments  of  participants. 

From  the  participants’  perspective,  one  of  the  most  difficult  tasks  of  the 
experiment  was  understanding  the  problem.  This  is  a  common  comment  for  experiments 
using  tactical  decision  making  tasks.  In  a  typical  staff  operation  setting  in  the  field,  staff 
officers  are  likely  to  have  been  involved  in  the  garrison  preparation  for  the  exercise.  In 
actual  combat,  officers  would  form  their  understanding  differently,  based  on  a  longer 
period  of  time  in  the  situation.  Understandings  would  develop  over  time  based  on 
changes  in  an  evolving  situation.  New  information  would  be  compared  to  the  last 
understanding  of  the  situation.  In  this  experiment,  participants  were  virtually  "thrown 
into  the  problem."  Although  the  scenario  dealt  with  combat  in  an  established  theater, 
the  rapid  understanding  required  of  staff  officers  might  be  more  typical  of  possible 
future,  rapid  response  conditions. 

All  i  articipants  who  were  asked  about  the  utility  of  the  structured  approach  felt 
that  it  was  nelpful  to  use  the  procedures  and  that  the  steps  were  consistent  with  current 
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instruction  in  CGSC.  For  the  structured  conditions,  the  work  sheets  were  considered  to 
be  useful  job  aids  in  their  own  right  for  organizing  and  analyzing  data.  Some 
participants  expressed  problems  in  providing  the  battle  outcome  predictions  which 
corresponded  to  their  low  confidence  ratings.  Current  U.S.  Army  procedures  and  data 
bases  for  making  outcome  predictions  were  considered  inadequate.  War  gaming  is 
considered  necessary,  but  it  is  difficult  and  time  consuming. 

Certain  computer  capabilities  (separate  from  those  used  in  this  experiment)  were 
considered  critical  by  participants.  Computer  tools  were  believed  to  be  necessary  for 
correlation  of  forces,  calculation  of  combat  power  ratios,  estimation  of  attrition, 
estimation  of  time  and  distance  parameters,  and  preparing  range  fans.  Other  functions 
would  include  display  of  check-lists  to  use  as  job  aids,  word  processing  capabilities,  and 
most  importantly,  a  printing  capability  to  provide  hard  copies  of  screen  displays  and 
computational  results. 

Participants  did  have  some  problems  with  computer  support.  Use  of  computer- 
displayed  maps  posed  problems  for  several  participants.  Familiarity  and  experience  with 
paper  maps  was  a  basis  for  this  problem.  In  addition,  the  limited  display  size  for  maps 
made  it  difficult  for  participants  to  see  the  "big  picture." 


Summary 

Unspecified  versus  Structured  Tnxedures 

Comparison  of  unspecified  and  structured  procedures  was  possible  by  observing 
the  procedures  used  by  the  unspecified  teams  compared  to  the  procedures  required  of 
the  structured  and  computer-supported  teams.  The  unspecified  teams  knew  about  the 
estimate  and  actively  sought  guidance  from  FM  101-5  and  ST  100-9.  There  were 
significant  differences  between  the  conditions  in  that  the  unspecified  teams,  individually 
or  as  a  group,  did  not  follow  a  systematic  series  of  steps.  The  most  systematic  team  from 
the  unspecified  condition  spent  almost  their  entire  time  calculating  combat  power  values 
and  ratios,  but  did  so  from  a  static  look  at  the  initial  dispositions.  They  did  not  do  war 
gaming.  The  quality  of  the  solutions  differed  with  the  unspecified  teams  doing 
significantly  poorer  than  teams  following  structured  procedures.  The  structured 
procedures  led  the  teams  to  produce  interim  results  which  distinguished  very  little 
between  the  COAs.  However,  the  procedures  appeared  to  force  the  structured  (manual 
and  computer-supported)  teams  to  do  a  detailed  enough  analysis  to  make  logical 
conclusions  about  the  relative  feasibility  of  the  two  COAs. 

Had  officers  with  greater  experience  in  tactical  planning  and  battlefield  operations 
participated,  it  is  envisioned  that  they  would  have  been  able  to  more  readily  see  the  risks 
and  opportunities  in  each  course  of  action.  It  is  likely  that  experts  would  not  (and  do 
not)  tolerate  a  slow,  rigid,  and  arduous  process  which  might  only  show  inconsequential 
differences  in  combat  outcomes  or  with  outcomes  that  are  largely  uncertain. 
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A  challenge  for  refining  estimate  procedures  is  to  find  a  balance  between  steps 
and  sequences  of  steps  which  can  be  used  to 

•  Apply  the  estimate  in  the  most  efficient  manner  possible, 

•  Be  systematic  enough  to  reduce  the  occurrence  of  groupthink  and  other  biases, 

•  Preclude  undertaking  steps  with  little  value,  and 

•  Avoid  over-quantifying  uncertain  events  and  outcomes. 


Manual  versus  Computer-supported  Implementation  of  Decision  Making  Procedures 

Identification  of  pertinent  facts,  arraying  of  the  forces,  critical  events,  and  war 
gaming  provided  a  means  to  compare  the  relative  effectiveness  of  manual  versus 
computer  use  of  prescribed  procedures  during  decision  making.  While  the  small  sample 
size  limits  the  certainty  and  generalizability  of  this  comparison,  the  data  suggest  that 
there  are  few  differences.  Both  structured  and  computer-supported  conditions  led  to 
similar  performance  on  identification  of  pertinent  facts,  arraying  forces,  critical  events, 
and  war  gaming.  Also  there  was  no  difference  in  solution  quality  between  the  two 
conditions. 

Thnse  in  the  computer-supported  condition  had  no  errors  in  arithmetic 
calculations  because  of  the  COAAT  tool,  while  most  of  the  structured  teams  made 
errors.  The  errors  were  not  severe  enough  to  make  a  difference  in  their  outcomes,  but  it 
is  likely  that  the  high  propensity  for  error  could  lead  to  incorrect  decisions  and  outcomes 
in  other  instances. 


Decision  and  Its  Justification 

Deciding  on  a  COA  in  combat  can  often  be  a  matter  of  determining  the  worst  of 
"evils".  In  this  experiment,  the  "most  critical"  event  for  COA  N  required  an  opposed 
river  crossing  across  an  unfordabie  river  at  night.  The  perils  of  this  situation  had  to  be 
carefully  considered  against  problems  associated  with  penetrating  a  numerically  superior 
enemy  for  COA  S. 


Observations  of  participants’  performance  suggest  that  decisions  were  often  based 
on  the  criticality  or  difficulty  of  the  actual  event,  and  not  the  quantitative  predictions. 
Along  this  line,  the  phrasing  of  subjective  factors  (e.g.,  risk)  provided  a  means  for 
participants  to  "pigeon  hole"  their  qualitative  observations  and  analyses  that  considered 
the  interaction  of  several  factors. 

Procedures  for  comparison  of  COAs  were  based  on  a  linear  model  that  generated 
aggregate  scores  for  both  objective  and  subjective  factors  (Steps  8  and  9).  The  utility  of 
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such  a  model  should  be  questioned.  The  river  crossing  operation  for  COA  N  is  an 
example  of  how  the  interaction  effect  of  many  factors  can  be  a  key  factor  in  the  decision. 
The  factors  of  terrain  (a  river),  environment  (night),  friendly  operations  (a  river 
crossing),  and  enemy  considerations  (a  dug-in  defensive  position)  combined  to  create  a 
difficult  and  complex  operation.  The  complexity  of  this  situation  makes  it  that  much 
harder  to  predict  battle  outcomes.  And  once  quantitative  projections  were  made,  it  was 
easy  for  the  predictions  of  a  single  event  to  get  "washed  out"  when  it  was  rolled  up  with 
other  events  to  produce  an  aggregate  score. 

Using  a  decision  analytic  approach,  as  complicated  as  a  weighted,  multi-attribute 
utility  matrix  or  as  simple  as  summary  columns  of  pluses  and  minuses,  can  be  misleading 
foi  complex,  dynamic  tactical  problems. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


This  experiment  examined  the  impact  of  procedural  structure  and  computer 
support  on  the  selection  of  a  tactical  course  of  action  by  Army  officers  role-playing  as 
division  plans  officers.  Experimental  results  and  comments  from  participants  indicated 
the  potential  benefits  of  using  a  structured  process  for  tactical  decision  making.  Using  a 
structured  approach  increased  the  likelihood  that  critical  factors  were  considered  in  the 
decision  making  process.  Even  when  the  structured  procedures  produced  a  marginally 
informative  outcome,  such  as  showing  little  or  no  distinction  in  objective  war  gaming 
scores,  going  through  the  process  helped  to  achieve  a  deeper  level  of  understanding  of 
the  tactical  operation  than  those  not  required  to  follow  structured  procedures. 

The  estimate  of  the  situation  has  been  undergoing  refinement  throughout  its  life 
(Michel,  1990).  The  format  of  the  estimate  in  1932  had  13  topical  headings.  In  1984  the 
format  grew  to  29  headings  with  additional,  separate  estimates  on  personnel,  intelligence, 
logistic,  civil  military  operations,  deception,  operations  security,  and  psychological 
operations.  The  results  of  this  experiment  indicate  that  refinement  to  procedures 
continues  to  be  a  viable  way  to  enhance  command  and  staff  operations.  The 
development  of  the  structured  condition  took  basic  concepts  from  standard  US  Army 
literature  on  the  estimate  and  proceduralized  them  into  a  workbook  of  instructions, 
guidelines,  checklists,  and  work  sheets.  These  materials  were  beneficial  to  the 
participants  in  the  structured  and  computer-supported  conditions.  Simple  job  aids  and 
work  sheets  can  be  an  enormous  benefit  in  helping  relatively  inexperienced  staff  consider 
multiple  factors  while  processing  information  foi  tactical  decisions. 


It  is  reasonable  to  expect  that  even  greater  gains  in  performance  can  be  made  if 
variations  in  procedures  are  explored  and  evaluated.  In  this  experiment  current 
procedures  were  incorporated  into  manual  instructions  and  job  aids  and  in  low-level 
computer  support.  The  typical  means  of  evaluating  or  testing  new  procedures  are  by 
staffing  proposed  written  changes  for  "desk-side"  review  or  trial  application  and 


subjective  assessment  by  staffs  in  the  field.  This  experiment  provided  an  alternative  to 
typical  means  of  evaluation  and  revealed  problems  with  the  estimate  that  had  been 
suspected  before,  but  for  which  firm  data  did  not  exist.  The  findings  have  in  turn 
suggested  enhancements. 


War  g<  ming  and  comparison  were  the  most  difficult  steps  in  the  experimental 
task.  Participants  generally  lacked  the  knowledge  and  experience  to  make  battle 
projections.  They  reported  that  neither  doctrine  or  instruction  provides  adequate 
guidance  for  making  war  gaming  projections.  Many  of  the  necessary  data  bases  are 
lacking,  especially  in  the  area  of  attrition.  More  comprehensive  data  bases  are  available 
for  measures  associated  with  time  and  distance  of  movement,  but  there  is  often  little 
time  for  planners  to  access  and  assimilate  this  information  manually  to  allow  application 
to  the  current  situation. 


One  strategy  of  planners  seems  to  be  to  avoid  making  projections.  Comparing  the 
structured  and  unspecified  conditions  suggests  that  when  planners  war  game,  even  though 
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the  war  gaming  does  not  identify  differences  or  if  there  are  highly  questionable 
outcomes,  the  planners  have  been  helped  by  trying  to  make  those  projections  to  better 
visualize  and  understand  the  COAs. 

Planning  might  well  benefit  from  automated  support  to  alleviate  the  analytical 
and  quantitative  burden  associated  with  projecting  battlefield  mtcomes.  Projection  is 
beyond  the  role  of  COAAT  with  it  spreadsheets  used  for  recording  and  tabulating 
projections.  Participants  in  the  current  experiment  expressed  a  need  for  support  to 
correlation  of  forces,  calculation  of  combat  power  ratios,  estimation  of  attrition,  and 
estimation  cf  time  and  distance  parameters.  These  computational  activities  can  be  more 
accurately  and  consistently  performed  by  computer  than  humans,  provided  that 
standardized  data  bases  and  war  gaming  algorithms  can  be  generated  and  accepted  by 
the  U.S.  Army  and  the  joint  services. 

Experimental  results  and  participant  feedback  indicate  that  procedures  and 
automated'  support  should  enhance  the  user’s  capability  to  identify  complex  interactive 
factors  that  can  impact  upon  the  success  or  failure  of  a  mission.  While  it  is  important  to 
thoroughly  understand  the  battlefield,  it  also  appears  promising  to  provide  user’s  with 
procedural  job  aids  and  automated  support.  Projecting  battle  results  and  enhancing 
battlefield  visualization  are  two  top  candidates  for  computer  support.  Computer  support 
tools  can  be  developed  that  will  enhance  the  user’s  capability  to  determine  and  analyze 
critical  battlefield  events  that  significantly  contribute  to  the  likelihood  of  mission  success 
or  failure. 


Also  it  is  important  to  recognize  that  the  computer  support  used  in  this 
experiment  has  functional  characteristics  and  features  that  can  facilitate  tactical  decision 
making.  In  fact,  the  task  organization  and  status  tool  (TOAST)  from  the  Tactical 
Planning  Workstation  has  been  used  as  a  model  for  a  similar  tool  in  a  field  tactical  C2 
system.  Also  based  on  the  findings  from  this  experiment,  the  Workstation  has  been 
subsequently  enhanced  with  additional  tools.  Capabilities  have  been  added  in  the  areas 


of  projecting  battlefield  outcomes  and  aiding  the  visualization  of  the  battlefield.  This 


suite  of  tools  called  Operations  Planning  Tools  (OPT)  has  undergone  preliminary  testing 
with  representative  users  and  has  met  with  high  levels  of  acceptance  (Fallesen,  1991a; 
Fallesen,  1991b;  Perkins,  Flanagan,  &  Fallesen,  1991). 


Planning  tools  and  decision  aids  like  these  offer  great  potential  to  commanders 
and  staffs.  Only  with  careful  examination  of  the  jobs,  knowledge,  and  styles  of  soldiers 
will  insightful  requirements  for  effective  tactical  planning  and  decision  aids  emerge. 
Regardless  of  whether  the  potential  for  enhancement  comes  from  decision  aids, 
improved  training,  or  enriched  organizations,  the  capabilities  and  limitations  of 
commanders  and  staffs  must  be  thoroughly  understood.  As  the  battlefield  grows  in 
complexity  and  military  forces  are  challenged  with  new  missions  and  roles,  the  need  for 
continuing  research  in  all  facets  of  command  and  control  remains  strong. 
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Appendix  A 

Background  Questionnaire 
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Appendix  B 

Experimental  Task  Procedures  Workbook 
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Analyzing  Tactical  Courses  ol  Action 


Figure  1.  Oecision-Making  Cycle 
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Candidate  Critical  Event  Types 


After  you  have  completed  the  CE  Identification  on  the  tactical  ovarleys  for  both  COAj 
and  have  completed  the  CE  Assignment  Worksheet,  you  have  completed  the  first  phase  ol  the 
exercise.  Additional  instructions  an*  be  provided  IP  you  at  this  time. 
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Scaling  Factors  Table  (Objective  Measures) 
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When  you  have  reviewed  the  battle  assessments  end  tea  hog  in  the  Wai-Gaming 
try  Worksheet.  you  may  proceed  to  Step  8. 
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Situation  Awareness  Questionnaire 


SITUATION  AWARENESS  QUESTIONNAIRE 

1.  Which  of  the  following  statements  best  describes  the  primary  purpose  of  the  mission  given  to 

the  16  MID? 

a.  destroy  enemy  in  zone 
(^seize  key  terrain 

c.  fix  enemy  in  position 

d.  turn  flank  of  10  CAA 

2.  Which  of  the  following  is  not  an  implied  task  for  the  mission,  of  the  16  MUD? 

a.  river  crossing 

b.  coordination  with  adjacent  units 
©destruction  of  bypassed  enemy  units 

d.  passage  of  lines 

'M 

3.  What  is  the  estimated  current  strength  of  the  15  MRD7 


a.  50% 

©60% 

c.  70% 

d.  80% 


4.  Which  of  the  following  15  MRD  regiments  is  equipped  with  BMPS? 

a.  45  MRR 

b.  46  MRR 
©47  MRR 

a.  all  of  the  above 

5.  How  many  maneuver  battalions  are  in  the  45  MRR? 

a.  2 

©3 

c.  4 

d.  5 

6.  Which  of  the  following  regiments  is  strongest  in  terms  of  estimated  percent  of  combat 

strength  remaining? 

©18  TR 

b.  45  MRR 

c.  46  MRR 

d.  47  MRR 
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7.  How  many  artillery  battalions  are  in  the  RAGs  of  each  of  the  forward  regiments  of  the  15 

MRD? 

a.  1 

©2 

c.  3 

d.  4 

8.  What,  is.  the  most  likely  mission  of  the  33  GTD? 

a.  10  CAA  second  echelon  defense  forces  i 

(Gi)  mobiles  reserve 

c,  bold  the  right  (north)  flank  of  the  10  CAA 

<;  relieve  the  15  MRD 

9.  Which  of  the  following  is  asi  a  current  weakness  of  the  enemy? 

a.  resupply 

b.  maneuver  strength 

c.  helicopter  strength 
(cl)  artillery  ammunition 

10.  What  is  the  most  likely  mission  of  the  10  CAA? 

a.  counterattack  to  disrupt  10  Corps  buildup 

b.  defend  in  position 

(£)  delay  back  to  the  Fulda  river 
o.  delay  back  to  the  ICIB 

11.  What  is  the  mix  of  mechanized  and  armored  battalions  in  the  16  MID? 

Mech  Armor 
a.  6  4 

i  x  ^ 

o.o  o 

©5  5 

d.  6  6 

12.  Which  of  the  following  16  MID  battalions  is  no!  currently  in  contact? 

a.  2-6  AR 
(2)2-10  AR 

c.  2-  71  INF 

d.  2-74  INF 

13.  How  many  ground  maneuver  battalions  are  currently  in  the  reserve  brigade  (2  BDE)? 

a.  1 

<E>2 

c.  3 

d.  4 


C-3 


14.  How  many  AH  battalions  are  currently  under  division  control  (not  counting  the  Armored 

Cav  Squadron)? 

a.  two  AH-1  bns 

b.  two  AH-1  bns  and  one  AH -64  bn 

c.  two  AN-64  bns  and  one  AH-1  bn 
(3)  three  AH-64  bns 

15.  What  is  the  total  number  of  155mm  howitzer  battalions  currently  assigned  or  attached  to  the 

division? 

1 

a.  3 

b.  4 

c.  5 

(3)6 

16.  Which  of  the  following  was  not  an  assumption/conclusion  contained  in  the  division  personnel 

estimate? 

a.  COA  2  can  best  be  supported  from  a  personnel  standpoint. 

Projected  replaced  rates  will  match  casualty  rates  over  the  mission  time  period. 

c.  Significant  combat  arms  replacements  will  be  required  upon  mission  completion. 

d.  Expected  daily  casualty  rate  will  not  exceed  5  percent, 

17.  Which  of  the  following  was  nal  an  assumption/condusion  contained  in  the  division  logistics 

estimate? 

COA  2  can  best  be  supported  from  a  logistics  standpoint. 

b.  Supplies  will  be  available  to  support  continued  offensive  operations  as  planned. 

c.  Shortage  of  cargo  transportation  in  maneuver  battalions  may  impact  combat  operations 
if  resupply  is  not  forthcoming. 

d.  Hard-surface  MSRs  are  available  to  support  either  COA. 

16.  How  many  CAS  sorties  has  the  16  MID  been  allocated  for  this  mission? 

32 

b.  42 

c.  48 

d.  56 

19.  Where  is  the  Fulda  River  most  fordable  in  the  division  AO? 

(a) south 

b.  central 

c.  north 

d.  equally  fordable  throughout 
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20.  What  is  the  approximate  distance  in  kilometers  from  the  current  FLOT  to  the  objective? 

a.  30 
<&40 

c.  50 

d.  60 

21.  Which  of  the  following  urban  areas  is  nfit  in  the  division  AO? 

a.  Lauterbach 

b.  Schlitz 

c.  Kunfeld  i 

(3)  Fulda 

22.  Approximately  how  many  kilometers  is  it  from  the  current  FLOT  to  the  Fulda  River  along 
avenue  ALPHA  (northern  avenue)? 

a.  10  ** 

b.  15 

©20 

d.  25 

23.  On  what  day  relative  to  the  mission  is  the  next  rain  expected? 

a.  first  day 

b.  second  day 
Othird  day 

d.  fourth  day 

24.  Wind  direction  for  the  next  two  days  will  be  generally  out  of  the: 

a.  north 

b.  south 

c.  east 
^cywest 

25.  Which  type  of  terrain  is  most  prevalent  in  the  division  AO  west  of  the  Fulda  River? 

a.  Go 

(^Slow-Go 

c.  No-Go 

d.  about  equal 

26.  How  many  times  will  the  division  have  to  cross  an  Autobahn  between  the  current  FLOT  and 
the  objective? 

a.  none 
(5)  once 

c.  twice 

d.  three  times 
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27.  Within  how  many  hours  of  H-Hour  (051200  Sep)  must  the  division  be  on  the  objective? 

a.  18 

b.  24 

c.  36 

(3)  not  in  the  corps  order 

28.  Relative  to  the  16  MID  attack,  when  is  the  32  MID  attack  to  begin? 

a.  4  hours  earlier 

^^simultaneously  , 

c.  4  hours  later 

d.  6  hours  later 

29.  Which  of  the  following  is  the  division  commander’s  preference  for  force  distribution  in  the 

attack? 

(&}  two  brigades  on  line  with  a  strong  reserve 
o.  two  brigades  on  line  with  a  small  reserve 

c.  all  three  brigades  on  line 

d.  no  preference  given 

30.  Which  of  the  following  was  not  a  special  concern  of  the-  division  commander  according  to  his 

guidance? 

a.  indications  of  possible  enemy  use  of  nuclear  weapons 

b.  indications  of  reinforcement  of  enemy  units  in  contact 
(^indications  of  counterattack  by  15  MRD 

d.  indications  of  movement  of  the  14  TA 

31.  The  32  AD  will  relieve  elements  of  the  16  MID  on  the  right  (south)  flank: 

a.  during  the  night  of  04/05  Sep 

b.  by  dawn,  05  Sep 

c.  prior  to  051200  Sep 

@  following  the  passage  of  lines  at  051200  Sep 

32.  Which  of  the  following  best  describes  the  mission  of  the  313  Sep  Mech  Bde? 

a.  act  as  corps  RACO  force  throughout  the  mission 
([b)  follow  16  MID,  be  prepared  for  commitment  in  zones  of  16  MID  and  32  AD  in  order 

c.  follow  32  AD,  be  prepared  for  commitment  in  zones  of  32  AD  and  16  MID  in  order 

d.  attached  to  12  MID  as  corps  reserve 
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COA  Analysis  Task  Evaluation 


_  ,  3.  Please  indicate  the  amount  of  affect  you  expended  performing  each  of  the  following  potential 

COA  ANALYSIS  TASK  EVALUA  I  ION  aclivitiee  as  part  ot  She  COA  anafy* I*  task.  Cirde  the  phrase  that  most  dosely  indicate*  the  an 


Characteristics 


How  difficult  was  it  to  perform  each  of  the  following  potential  ecffvitlea  at  pert  of  the  CCA  analysis 
task?  Please  circle  the  phrase  that  most  closely  indicate*  the  degree  of  difficulty.  Oont  antvrer  for 
an  activity  for  which  no  effort  wee  wpended  (Question  3  above) 
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3  How  difficult  w*»  K  to  perfom)  each  et  th«  >tops  ot  COK  analysis  1W*d  batow?  Plsase  »e!« 

COA  ANALYSIS  TASK  EVALUATION  p*""  "“•*  dMrt»  MSe*"  •*  d*°t"  01  <sfficuny 
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Human-Machine  Interface  Evaluation 
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Team  Profile  Work  Sheet 


checking  dam 


TEAM  PROFILE 


